I agree that we can start from Pulsar sink and Pulsar Catalog. When we finish that, maybe FLIP-27 is ready then.
Best, Jark On Tue, 24 Sep 2019 at 17:06, Becket Qin <becket....@gmail.com> wrote: > Hi Till, > > My understanding is that we are going to hold back a little bit on the > Pulsar source connector until FLIP-27 is ready. If we check in the old > Pulsar source connector right away, we will have to maintain that old > connector for quite a while (a year or more), which is something we don't > want to do. > > So the conclusion is following: > - We will try our best to check in both FLIP-27 and the Pulsar source > connector in 1.10. > - In case there isn't enough time to get Pulsar source connector checked in > 1.10, we will link prominently to the external Pulsar connector repo in our > connector doc so that users can find that easily. > > BTW, I think we can start the work on Pulsar sink connector given that > there is no clear plan on whether or how we are going to change the sink > interface. > > @Sijie, @Yijie, what do you think? > > Thanks, > > Jiangjie (Becket) Qin > > On Tue, Sep 24, 2019 at 4:40 PM Till Rohrmann <trohrm...@apache.org> > wrote: > > > What is the conclusion of this discussion? Are we gonna merge the old > > source/sink or do we believe that FLIP-27 will be completed in time so > that > > we can merge the new source right away? > > > > Cheers, > > Till > > > > On Tue, Sep 24, 2019 at 3:43 AM 郭士榕 <gsra...@163.com> wrote: > > > > > > > > > > > unscribe > > > > > > > > > > > > > > > > > > > > > At 2019-09-24 08:53:25, "Becket Qin" <becket....@gmail.com> wrote: > > > >Thanks, Stephan. > > > > > > > >Sounds good to me. We can still try our best to get new Pulsar > connector > > > in > > > >Flink 1.10. In case we do not have time to do that, we will > prominently > > > >link the Pulsar connector from the Flink connector docs. > > > > > > > >Thanks, > > > > > > > >Jiangjie (Becket) Qin > > > > > > > >On Mon, Sep 23, 2019 at 4:11 PM Stephan Ewen <se...@apache.org> > wrote: > > > > > > > >> Okay, I see your point, Becket. > > > >> > > > >> Then let us prominently link the Pulsar connector from the Flink > > > connector > > > >> docs then, so that users can find it easily. > > > >> > > > >> As soon as FLIP 27 is done, we reach out the Pulsar folks to > > contribute > > > a > > > >> new connector. > > > >> > > > >> On Mon, Sep 23, 2019 at 3:11 AM Becket Qin <becket....@gmail.com> > > > wrote: > > > >> > > > >> > Hi Stephan, > > > >> > > > > >> > I have no doubt about the value of adding Pulsar connector to > Flink > > > repo. > > > >> > My concern is about how exactly we are going to do it. > > > >> > > > > >> > As mentioned before, I believe that we can handle connectors more > > > >> > > pragmatically and less strict than the core of Flink, if it > helps > > > >> > unlocking > > > >> > > users. > > > >> > > > > >> > I can see the benefit of being less restrict for the initial > > connector > > > >> code > > > >> > adoption. However, I don't think we should be less restrict on the > > > >> > maintenance commitment once the code is in Flink repo. It only > makes > > > >> sense > > > >> > to check in something and ask users to use if we plan to maintain > > it. > > > >> > > > > >> > If I understand correctly, the current plan so far is following: > > > >> > 1. release 1.10 > > > >> > - Check in Pulsar connector on old interface and label it as > beta > > > >> > version. > > > >> > - encourage users to try it and report bugs. > > > >> > 2. release 1.11 > > > >> > - Check in Pulsar connector on new interface (a.k.a new Pulsar > > > >> > connector) and label it as beta version > > > >> > - Deprecate the old Pulsar connector > > > >> > - Fix bugs reported on old Pulsar connector from release 1.10 > > > >> > - Ask users to migrate from old Pulsar connector to new Pulsar > > > >> connector > > > >> > 3. release 1.12 > > > >> > - Announce end of support for old Pulsar connector and remove > the > > > code > > > >> > - Fix bugs reported on new Pulsar connector. > > > >> > > > > >> > If this is the plan, it seems neither Flink nor the users trying > the > > > old > > > >> > Pulsar connector will benefit from this experimental old Pulsar > > > >> connector, > > > >> > because whatever feedbacks we got or bugs we fix on the old Pulsar > > > >> > connector are immediately thrown away in one or two releases. > > > >> > > > > >> > If we check in the old Pulsar connector right now, the only > option I > > > see > > > >> is > > > >> > to maintain it for a while (e.g. a year or more). IMO, the > immediate > > > >> > deprecation and code removal hurts the users much more than asking > > > them > > > >> to > > > >> > wait for another release. I personally think that we can avoid > this > > > >> > maintenance burden by going directly to the new Pulsar connector, > > > >> > especially given that users can still use the connector even if > they > > > are > > > >> > not in Flink repo. That said, I am OK with maintaining both old > and > > > new > > > >> > Pulsar connector if we believe that having the Pulsar connector > > > available > > > >> > right now in Flink repo is more important. > > > >> > > > > >> > Thanks, > > > >> > > > > >> > Jiangjie (Becket) Qin > > > >> > > > > >> > On Sun, Sep 22, 2019 at 9:10 PM Stephan Ewen <se...@apache.org> > > > wrote: > > > >> > > > > >> > > My assumption is as Sijie's, that once the connector is either > > part > > > of > > > >> > > Flink, or part of the streamnative repo. No double maintenance. > > > >> > > > > > >> > > I feel this discussion is very much caught in problems that are > > all > > > >> > > solvable if we want to solve them. > > > >> > > Maybe we can think what our goal for users and the communities > is? > > > >> > > > > > >> > > - Do we want to help build a relationship between the Pulsar > and > > > >> Flink > > > >> > > open source communities? > > > >> > > - Will users find a connector in the streamnative repository? > > > >> > > - Will users trust a connector that is not part of Flink as > > much? > > > >> > > > > > >> > > And then decide what is best according to the overall goals > there. > > > >> > > As mentioned before, I believe that we can handle connectors > more > > > >> > > pragmatically and less strict than the core of Flink, if it > helps > > > >> > unlocking > > > >> > > users. > > > >> > > > > > >> > > Best, > > > >> > > Stephan > > > >> > > > > > >> > > > > > >> > > > > > >> > > On Fri, Sep 20, 2019 at 2:10 PM Sijie Guo <guosi...@gmail.com> > > > wrote: > > > >> > > > > > >> > > > Thanks Becket. > > > >> > > > > > > >> > > > I think it is better for the Flink community to judge the > > > benefits of > > > >> > > doing > > > >> > > > this. I was trying to provide some views from outsiders. > > > >> > > > > > > >> > > > Thanks, > > > >> > > > Sijie > > > >> > > > > > > >> > > > On Fri, Sep 20, 2019 at 10:25 AM Becket Qin < > > becket....@gmail.com > > > > > > > >> > > wrote: > > > >> > > > > > > >> > > > > Hi Sijie, > > > >> > > > > > > > >> > > > > Yes, we will have to support existing old connectors and new > > > >> > connectors > > > >> > > > in > > > >> > > > > parallel for a while. We have to take that maintenance > > overhead > > > >> > because > > > >> > > > > existing connectors have been used by the users for a long > > > time. I > > > >> > > guess > > > >> > > > It > > > >> > > > > may take at least a year for us to fully remove the old > > > connectors. > > > >> > > > > > > > >> > > > > Process wise, we can do the same for Pulsar connector. But I > > am > > > not > > > >> > > sure > > > >> > > > if > > > >> > > > > we want to have the same burden on Pulsar connector, and I > > would > > > >> like > > > >> > > to > > > >> > > > > understand the benefit of doing that. > > > >> > > > > > > > >> > > > > For users, the benefit of having the old Pulsar connector > > > checked > > > >> in > > > >> > > > seems > > > >> > > > > limited because 1) that code base will be immediately > > > deprecated in > > > >> > the > > > >> > > > > next release in 3-4 months; 2) users can always use it even > if > > > it > > > >> is > > > >> > > not > > > >> > > > in > > > >> > > > > the Flink code base. Admittedly it is not as convenient as > > > having > > > >> it > > > >> > in > > > >> > > > > Flink code base, but doesn't seem super either. And after > 3-4 > > > >> months, > > > >> > > > users > > > >> > > > > can just use the new connector in Flink repo. > > > >> > > > > > > > >> > > > > For Flink developers, the old connector code base is not > > > something > > > >> > that > > > >> > > > we > > > >> > > > > want to evolve later. Instead, these code will be deprecated > > and > > > >> > > > > removed. So why do we want to get a beta version out to > > attract > > > >> > people > > > >> > > to > > > >> > > > > use something we don't want to maintain? > > > >> > > > > > > > >> > > > > Thanks, > > > >> > > > > > > > >> > > > > Jiangjie (Becket) Qin > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > On Fri, Sep 20, 2019 at 10:12 AM Sijie Guo < > > guosi...@gmail.com> > > > >> > wrote: > > > >> > > > > > > > >> > > > > > Thanks everyone here. Sorry for jumping into the > discussion > > > here. > > > >> > > > > > > > > >> > > > > > I am not very familiar about the deprecation process in > > Flink. > > > >> If I > > > >> > > > > > misunderstood the process, please fix me. > > > >> > > > > > > > > >> > > > > > As far as I understand, FLIP-27 is introducing a new > unified > > > API > > > >> > for > > > >> > > > > > connectors. After it introduces the new API > > > >> > > > > > and before moving all the existing connectors from old API > > to > > > new > > > >> > > API, > > > >> > > > > both > > > >> > > > > > old ApI and new API will co-exist > > > >> > > > > > for a while until Flink moves all existing connectors to > new > > > API. > > > >> > So > > > >> > > > the > > > >> > > > > > Pulsar connector (using old API) can > > > >> > > > > > follow the deprecation process with other connector using > > old > > > API > > > >> > and > > > >> > > > the > > > >> > > > > > deprecation of old API, no? > > > >> > > > > > > > > >> > > > > > If that's the case, I think contributing the current > > connector > > > >> back > > > >> > > to > > > >> > > > > > Flink rather than maintaining it outside Flink > > > >> > > > > > would provide a bit more benefits. We can deprecate the > > > existing > > > >> > > > > > streamnative/pulsar-flink repo and point the users > > > >> > > > > > to use the connector in Flink repo. So all the review > > > processes > > > >> > will > > > >> > > > > happen > > > >> > > > > > within Flink for both old connector and > > > >> > > > > > new connector. It also reduces the confusions for the > users > > as > > > >> the > > > >> > > > > > documentation and code base happen in one place. > > > >> > > > > > > > > >> > > > > > Thoughts? > > > >> > > > > > > > > >> > > > > > - Sijie > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > On Fri, Sep 20, 2019 at 12:53 AM Becket Qin < > > > >> becket....@gmail.com> > > > >> > > > > wrote: > > > >> > > > > > > > > >> > > > > > > Thanks for the explanation, Stephan. I have a few > > questions > > > / > > > >> > > > thoughts. > > > >> > > > > > > > > > >> > > > > > > So that means we will remove the old connector without a > > > major > > > >> > > > version > > > >> > > > > > > bump, is that correct? > > > >> > > > > > > > > > >> > > > > > > I am not 100% sure if mixing 1.10 connectors with 1.11 > > > >> connectors > > > >> > > > will > > > >> > > > > > > always work because we saw some dependency class > > collisions > > > in > > > >> > the > > > >> > > > > past. > > > >> > > > > > To > > > >> > > > > > > make it safe we may have to maintain the old code for > one > > > more > > > >> > > > release. > > > >> > > > > > > > > > >> > > > > > > To be honest I am still wondering if we have to put the > > old > > > >> > > connector > > > >> > > > > in > > > >> > > > > > > Flink repo. if we check in the old connector to Flink. > We > > > will > > > >> > end > > > >> > > up > > > >> > > > > in > > > >> > > > > > > the following situation: > > > >> > > > > > > 1. Old connector in streamnative/pulsar-flink repo. > > > >> > > > > > > 2. Old connector in Flink Repo, which may be different > > from > > > the > > > >> > one > > > >> > > > in > > > >> > > > > > > Pulsar repo. (Added in 1.10, deprecated in 1.11, removed > > in > > > >> 1.12) > > > >> > > > > > > 3. New connector in Flink Repo. > > > >> > > > > > > > > > >> > > > > > > We need to think about how to make the users in each > case > > > >> happy. > > > >> > > > > > > - For users of (1), I assume Sijie and Yijie will have > to > > > >> > maintain > > > >> > > > the > > > >> > > > > > code > > > >> > > > > > > a bit longer for its own compatibility even after we > have > > > (2). > > > >> In > > > >> > > > that > > > >> > > > > > > case, bugs found in old connector may or may not need to > > be > > > >> fixed > > > >> > > in > > > >> > > > > both > > > >> > > > > > > Flink and the streamnative/pulsar-flink repo. > > > >> > > > > > > - For users of (2), will we provide bug fixes? If we do, > > it > > > >> will > > > >> > > be a > > > >> > > > > > > little awkward because those bug fixes will be > immediately > > > >> > > deprecated > > > >> > > > > in > > > >> > > > > > > 1.11, and removed in 1.12. So we are essentially asking > > > users > > > >> to > > > >> > > > > migrate > > > >> > > > > > > away from the bug fix. After Flink 1.12, users may still > > > have > > > >> to > > > >> > > > switch > > > >> > > > > > to > > > >> > > > > > > use (3) due to the potential dependency class conflicts > > > >> mentioned > > > >> > > > > above. > > > >> > > > > > > - Users of (3) have a much easier life and don't need to > > > worry > > > >> > too > > > >> > > > > much. > > > >> > > > > > > > > > >> > > > > > > The above story seems a little complicated to tell. I > > think > > > it > > > >> > will > > > >> > > > be > > > >> > > > > > much > > > >> > > > > > > easier to not have (2) at all. > > > >> > > > > > > 1. Old connector in streamnative/pulsar-flink repo. > > > >> > > > > > > 3. New connector in Flink Repo. > > > >> > > > > > > > > > >> > > > > > > - Old connector will only be maintained in > > > >> > > streamnative/pulsar-flink > > > >> > > > > repo > > > >> > > > > > > until it is fully deprecated. Users can always use the > > > existing > > > >> > > > Pulsar > > > >> > > > > > > connector in that repo. > > > >> > > > > > > - New connector will be in Flink repo and maintained > like > > > the > > > >> > other > > > >> > > > > > > connectors. > > > >> > > > > > > > > > >> > > > > > > This seems much simpler for users understand and they > will > > > not > > > >> be > > > >> > > > > blocked > > > >> > > > > > > from using the old connector. If the concern is about > the > > > >> quality > > > >> > > of > > > >> > > > > the > > > >> > > > > > > connector in streamnative/pulsar-flink repo, is it > enough > > > for > > > >> us > > > >> > > just > > > >> > > > > to > > > >> > > > > > > review the code in streamnative/pulsar-flink connector > to > > > make > > > >> > sure > > > >> > > > it > > > >> > > > > > > looks good from Flink's perspective? > > > >> > > > > > > > > > >> > > > > > > What do you think? > > > >> > > > > > > > > > >> > > > > > > Thanks, > > > >> > > > > > > > > > >> > > > > > > Jiangjie (Becket) Qin > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > On Thu, Sep 19, 2019 at 6:58 PM Stephan Ewen < > > > se...@apache.org > > > >> > > > > >> > > > wrote: > > > >> > > > > > > > > > >> > > > > > > > My take would be the following: > > > >> > > > > > > > > > > >> > > > > > > > - If we merge the connector now and replace it with > a > > > >> FLIP-27 > > > >> > > > > version > > > >> > > > > > > > before the 1.10 release, then we need no deprecation > > > process > > > >> > > > > > > > - If we don't manage to replace it with a FLIP-27 > > > version > > > >> > > before > > > >> > > > > the > > > >> > > > > > > 1.10 > > > >> > > > > > > > release, than it is good that we have the other > version, > > > so > > > >> no > > > >> > > > users > > > >> > > > > > get > > > >> > > > > > > > blocked. > > > >> > > > > > > > > > > >> > > > > > > > In the latter case we can see how we want to do it. > > > Immediate > > > >> > > > removal > > > >> > > > > > of > > > >> > > > > > > > the old version or deprecation label and keeping it > for > > > one > > > >> > more > > > >> > > > > > release. > > > >> > > > > > > > Given that you should be able to use a Flink 1.10 > > > connector > > > >> > with > > > >> > > > > Flink > > > >> > > > > > > 1.11 > > > >> > > > > > > > as well (stable public APIs) there is also a > workaround > > if > > > >> you > > > >> > > need > > > >> > > > > an > > > >> > > > > > > old > > > >> > > > > > > > connector in a newer version. So immediate removal > might > > > even > > > >> > be > > > >> > > > > > > feasible. > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > On Thu, Sep 19, 2019 at 11:09 AM Becket Qin < > > > >> > > becket....@gmail.com> > > > >> > > > > > > wrote: > > > >> > > > > > > > > > > >> > > > > > > > > Hi Stephan, > > > >> > > > > > > > > > > > >> > > > > > > > > Thanks for the clarification. I completely agree > with > > > you > > > >> and > > > >> > > > > Thomas > > > >> > > > > > on > > > >> > > > > > > > the > > > >> > > > > > > > > process of adding connectors to Flink repo. > However, I > > > am > > > >> > > > wondering > > > >> > > > > > > what > > > >> > > > > > > > is > > > >> > > > > > > > > the deprecation process? Given the main concern here > > was > > > >> that > > > >> > > we > > > >> > > > > may > > > >> > > > > > > have > > > >> > > > > > > > > to maintain two Pulsar connector code bases until > the > > > old > > > >> one > > > >> > > is > > > >> > > > > > > removed > > > >> > > > > > > > > from the repo, it would be good to know how long we > > > have to > > > >> > do > > > >> > > > > that. > > > >> > > > > > > > > > > > >> > > > > > > > > Thanks, > > > >> > > > > > > > > > > > >> > > > > > > > > Jiangjie (Becket) Qin > > > >> > > > > > > > > > > > >> > > > > > > > > On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen < > > > >> > se...@apache.org > > > >> > > > > > > >> > > > > > wrote: > > > >> > > > > > > > > > > > >> > > > > > > > > > Some quick thoughts on the connector contribution > > > >> process. > > > >> > I > > > >> > > > > > > basically > > > >> > > > > > > > > > reiterate here what Thomas mentioned in another > > thread > > > >> > about > > > >> > > > the > > > >> > > > > > > > Kinesis > > > >> > > > > > > > > > connector. > > > >> > > > > > > > > > > > > >> > > > > > > > > > For connectors, we should favor a low-overhead > > > >> contribution > > > >> > > > > > process, > > > >> > > > > > > > and > > > >> > > > > > > > > > accept user code and changes more readily than in > > the > > > >> core > > > >> > > > > system. > > > >> > > > > > > > > > That is because connectors have both a big variety > > of > > > >> > > scenarios > > > >> > > > > > they > > > >> > > > > > > > get > > > >> > > > > > > > > > used in (only through use and many small > > > contributions do > > > >> > > they > > > >> > > > > > become > > > >> > > > > > > > > > really useful over time) and at the same time, and > > > >> > committers > > > >> > > > do > > > >> > > > > > not > > > >> > > > > > > > use > > > >> > > > > > > > > > the connector themselves and usually cannot > foresee > > > too > > > >> > well > > > >> > > > what > > > >> > > > > > is > > > >> > > > > > > > > > needed. > > > >> > > > > > > > > > > > > >> > > > > > > > > > Further more, a missing connector (or connector > > > feature) > > > >> is > > > >> > > > > often a > > > >> > > > > > > > > bigger > > > >> > > > > > > > > > show stopper for users than a missing API or > system > > > >> > feature. > > > >> > > > > > > > > > > > > >> > > > > > > > > > Along these lines of thougt, the conclusion would > be > > > to > > > >> > take > > > >> > > > the > > > >> > > > > > > Pulsar > > > >> > > > > > > > > > connector now, focus the review on > > > >> legal/dependencies/rough > > > >> > > > code > > > >> > > > > > > style > > > >> > > > > > > > > and > > > >> > > > > > > > > > conventions, label it as "beta" (in the sense of > > "new > > > >> code" > > > >> > > > that > > > >> > > > > is > > > >> > > > > > > > "not > > > >> > > > > > > > > > yet tested through longer use") and go ahead. And > > then > > > >> > evolve > > > >> > > > it > > > >> > > > > > > > quickly > > > >> > > > > > > > > > without putting formal blockers in the way, > meaning > > > also > > > >> > > > adding a > > > >> > > > > > new > > > >> > > > > > > > > FLIP > > > >> > > > > > > > > > 27 version when it is there. > > > >> > > > > > > > > > > > > >> > > > > > > > > > Best, > > > >> > > > > > > > > > Stephan > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin < > > > >> > > > becket....@gmail.com > > > >> > > > > > > > > >> > > > > > > > wrote: > > > >> > > > > > > > > > > > > >> > > > > > > > > > > Hi Yijie, > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > Could you please follow the FLIP process to > start > > a > > > new > > > >> > > FLIP > > > >> > > > > > > > > [DISCUSSION] > > > >> > > > > > > > > > > thread in the mailing list? > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > I see two FLIP-69 discussion in the mailing list > > > now. > > > >> So > > > >> > > > there > > > >> > > > > > is a > > > >> > > > > > > > > FLIP > > > >> > > > > > > > > > > number collision. Can you change the FLIP number > > to > > > 72? > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > Thanks, > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > Jiangjie (Becket) Qin > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong < > > > >> > > > > walter...@gmail.com> > > > >> > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > Hi Yijie, > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > Thanks for sharing the pulsar FLIP. > > > >> > > > > > > > > > > > Would you mind enabling comments/suggestions > on > > > the > > > >> > > google > > > >> > > > > doc > > > >> > > > > > > > link? > > > >> > > > > > > > > > This > > > >> > > > > > > > > > > > way the contributors from the community can > > > comment > > > >> on > > > >> > > the > > > >> > > > > doc. > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > Best, > > > >> > > > > > > > > > > > Rong > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen < > > > >> > > > > > > > > henry.yijies...@gmail.com> > > > >> > > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > Hello everyone, > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > I've drafted a FLIP that describes the > current > > > >> design > > > >> > > of > > > >> > > > > the > > > >> > > > > > > > Pulsar > > > >> > > > > > > > > > > > > connector: > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit# > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > Please take a look and let me know what you > > > think. > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > Thanks, > > > >> > > > > > > > > > > > > Yijie > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong < > > > >> > > > > > > walter...@gmail.com> > > > >> > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Hi All, > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Sorry for joining the discussion late and > > > thanks > > > >> > > Yijie > > > >> > > > & > > > >> > > > > > > Sijie > > > >> > > > > > > > > for > > > >> > > > > > > > > > > > > driving > > > >> > > > > > > > > > > > > > the discussion. > > > >> > > > > > > > > > > > > > I also think the Pulsar connector would > be a > > > very > > > >> > > > > valuable > > > >> > > > > > > > > addition > > > >> > > > > > > > > > > to > > > >> > > > > > > > > > > > > > Flink. I can also help out a bit on the > > review > > > >> side > > > >> > > :-) > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Regarding the timeline, I also share > > concerns > > > >> with > > > >> > > > Becket > > > >> > > > > > on > > > >> > > > > > > > the > > > >> > > > > > > > > > > > > > relationship between the new Pulsar > > connector > > > and > > > >> > > > > FLIP-27. > > > >> > > > > > > > > > > > > > There's also another discussion just > started > > > by > > > >> > > Stephan > > > >> > > > > on > > > >> > > > > > > > > dropping > > > >> > > > > > > > > > > > Kafka > > > >> > > > > > > > > > > > > > 9/10 support on next Flink release [1]. > > > Although > > > >> > the > > > >> > > > > > > situation > > > >> > > > > > > > > is > > > >> > > > > > > > > > > > > somewhat > > > >> > > > > > > > > > > > > > different, and Kafka 9/10 connector has > been > > > in > > > >> > Flink > > > >> > > > for > > > >> > > > > > > > almost > > > >> > > > > > > > > > 3-4 > > > >> > > > > > > > > > > > > years, > > > >> > > > > > > > > > > > > > based on the discussion I am not sure if a > > > major > > > >> > > > version > > > >> > > > > > > > release > > > >> > > > > > > > > > is a > > > >> > > > > > > > > > > > > > requirement for removing old connector > > > supports. > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > I think there shouldn't be a blocker if we > > > agree > > > >> > the > > > >> > > > old > > > >> > > > > > > > > connector > > > >> > > > > > > > > > > will > > > >> > > > > > > > > > > > > be > > > >> > > > > > > > > > > > > > removed once FLIP-27 based Pulsar > connector > > is > > > >> > there. > > > >> > > > As > > > >> > > > > > > > Stephan > > > >> > > > > > > > > > > > stated, > > > >> > > > > > > > > > > > > it > > > >> > > > > > > > > > > > > > is easier to contribute the source sooner > > and > > > >> > adjust > > > >> > > it > > > >> > > > > > > later. > > > >> > > > > > > > > > > > > > We should also ensure we clearly > communicate > > > the > > > >> > > > message: > > > >> > > > > > for > > > >> > > > > > > > > > > example, > > > >> > > > > > > > > > > > > > putting an experimental flag on the > > pre-FLIP27 > > > >> > > > connector > > > >> > > > > > page > > > >> > > > > > > > of > > > >> > > > > > > > > > the > > > >> > > > > > > > > > > > > > website, documentations, etc. Any other > > > thoughts? > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > -- > > > >> > > > > > > > > > > > > > Rong > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > [1] > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket > Qin < > > > >> > > > > > > > becket....@gmail.com > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Technically speaking, removing the old > > > >> connector > > > >> > > code > > > >> > > > > is > > > >> > > > > > a > > > >> > > > > > > > > > > backwards > > > >> > > > > > > > > > > > > > > incompatible change which requires a > major > > > >> > version > > > >> > > > > bump, > > > >> > > > > > > i.e. > > > >> > > > > > > > > > Flink > > > >> > > > > > > > > > > > > 2.x. > > > >> > > > > > > > > > > > > > > Given that we don't have a clear plan on > > > when > > > >> to > > > >> > > have > > > >> > > > > the > > > >> > > > > > > > next > > > >> > > > > > > > > > > major > > > >> > > > > > > > > > > > > > > version release, it seems unclear how > long > > > the > > > >> > old > > > >> > > > > > > connector > > > >> > > > > > > > > code > > > >> > > > > > > > > > > > will > > > >> > > > > > > > > > > > > be > > > >> > > > > > > > > > > > > > > there if we check it in right now. Or > will > > > we > > > >> > > remove > > > >> > > > > the > > > >> > > > > > > old > > > >> > > > > > > > > > > > connector > > > >> > > > > > > > > > > > > > > without a major version bump? In any > case, > > > it > > > >> > > sounds > > > >> > > > > not > > > >> > > > > > > > quite > > > >> > > > > > > > > > user > > > >> > > > > > > > > > > > > > > friendly to the those who might use the > > old > > > >> > Pulsar > > > >> > > > > > > > connector. I > > > >> > > > > > > > > > am > > > >> > > > > > > > > > > > not > > > >> > > > > > > > > > > > > sure > > > >> > > > > > > > > > > > > > > if it is worth these potential problems > in > > > >> order > > > >> > to > > > >> > > > > have > > > >> > > > > > > the > > > >> > > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > source > > > >> > > > > > > > > > > > > > > connector checked in one or two months > > > earlier. > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Thanks, > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan > > > Ewen < > > > >> > > > > > > > se...@apache.org > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > Agreed, if we check in the old code, > we > > > >> should > > > >> > > make > > > >> > > > > it > > > >> > > > > > > > clear > > > >> > > > > > > > > > that > > > >> > > > > > > > > > > > it > > > >> > > > > > > > > > > > > will > > > >> > > > > > > > > > > > > > > > be removed as soon as the FLIP-27 > based > > > >> version > > > >> > > of > > > >> > > > > the > > > >> > > > > > > > > > connector > > > >> > > > > > > > > > > is > > > >> > > > > > > > > > > > > > > there. > > > >> > > > > > > > > > > > > > > > We should not commit to maintaining > the > > > old > > > >> > > > version, > > > >> > > > > > that > > > >> > > > > > > > > would > > > >> > > > > > > > > > > be > > > >> > > > > > > > > > > > > indeed > > > >> > > > > > > > > > > > > > > > too much overhead. > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket > > > Qin < > > > >> > > > > > > > > > becket....@gmail.com > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Hi Stephan, > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Thanks for the volunteering to help. > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Yes, the overhead would just be > review > > > >> > > capacity. > > > >> > > > In > > > >> > > > > > > > fact, I > > > >> > > > > > > > > > am > > > >> > > > > > > > > > > > not > > > >> > > > > > > > > > > > > > > > worrying > > > >> > > > > > > > > > > > > > > > > too much about the review capacity. > > > That is > > > >> > > just > > > >> > > > a > > > >> > > > > > one > > > >> > > > > > > > time > > > >> > > > > > > > > > > cost. > > > >> > > > > > > > > > > > > My > > > >> > > > > > > > > > > > > > > > > concern is mainly about the long > term > > > >> burden. > > > >> > > > > Assume > > > >> > > > > > we > > > >> > > > > > > > > have > > > >> > > > > > > > > > > new > > > >> > > > > > > > > > > > > source > > > >> > > > > > > > > > > > > > > > > interface ready in 1.10 with newly > > added > > > >> > Pulsar > > > >> > > > > > > > connectors > > > >> > > > > > > > > in > > > >> > > > > > > > > > > old > > > >> > > > > > > > > > > > > > > > > interface. Later on if we migrate > > > Pulsar to > > > >> > new > > > >> > > > > > source > > > >> > > > > > > > > > > interface, > > > >> > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > old > > > >> > > > > > > > > > > > > > > > > Pulsar connector might be deprecated > > > almost > > > >> > > > > > immediately > > > >> > > > > > > > > after > > > >> > > > > > > > > > > > > checked > > > >> > > > > > > > > > > > > > > in, > > > >> > > > > > > > > > > > > > > > > but we may still have to maintain > two > > > code > > > >> > > bases. > > > >> > > > > For > > > >> > > > > > > the > > > >> > > > > > > > > > > > existing > > > >> > > > > > > > > > > > > > > > > connectors, we have to do that > > anyways. > > > But > > > >> > it > > > >> > > > > would > > > >> > > > > > be > > > >> > > > > > > > > good > > > >> > > > > > > > > > to > > > >> > > > > > > > > > > > > avoid > > > >> > > > > > > > > > > > > > > > > introducing a new connector with the > > > same > > > >> > > > problem. > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Thanks, > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM > > Stephan > > > >> Ewen > > > >> > < > > > >> > > > > > > > > > se...@apache.org > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Hi all! > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Nice to see this lively discussion > > > about > > > >> > the > > > >> > > > > Pulsar > > > >> > > > > > > > > > > connector. > > > >> > > > > > > > > > > > > > > > > > Some thoughts on the open > questions: > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > ## Contribute to Flink or maintain > > as > > > a > > > >> > > > community > > > >> > > > > > > > package > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Looks like the discussion is more > > > going > > > >> > > towards > > > >> > > > > > > > > > > contribution. I > > > >> > > > > > > > > > > > > think > > > >> > > > > > > > > > > > > > > > > that > > > >> > > > > > > > > > > > > > > > > > is good, especially if we think > that > > > we > > > >> > want > > > >> > > to > > > >> > > > > > > build a > > > >> > > > > > > > > > > > similarly > > > >> > > > > > > > > > > > > > > deep > > > >> > > > > > > > > > > > > > > > > > integration with Pulsar as we have > > for > > > >> > > example > > > >> > > > > with > > > >> > > > > > > > > Kafka. > > > >> > > > > > > > > > > The > > > >> > > > > > > > > > > > > > > > connector > > > >> > > > > > > > > > > > > > > > > > already looks like a more thorough > > > >> > connector > > > >> > > > than > > > >> > > > > > > many > > > >> > > > > > > > > > others > > > >> > > > > > > > > > > > we > > > >> > > > > > > > > > > > > have > > > >> > > > > > > > > > > > > > > > in > > > >> > > > > > > > > > > > > > > > > > the repository. > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > With either a repo split, or the > new > > > >> build > > > >> > > > > system, > > > >> > > > > > I > > > >> > > > > > > > hope > > > >> > > > > > > > > > > that > > > >> > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > build > > > >> > > > > > > > > > > > > > > > > > overhead is not a problem. > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > ## Committer Support > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Becket offered some help already, > I > > > can > > > >> > also > > > >> > > > > help a > > > >> > > > > > > > bit. > > > >> > > > > > > > > I > > > >> > > > > > > > > > > hope > > > >> > > > > > > > > > > > > that > > > >> > > > > > > > > > > > > > > > > > between us, we can cover this. > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > ## Contribute now, or wait for > > FLIP-27 > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > As Becket said, FLIP-27 is > actually > > > >> making > > > >> > > some > > > >> > > > > > > PoC-ing > > > >> > > > > > > > > > > > > progress, but > > > >> > > > > > > > > > > > > > > > > will > > > >> > > > > > > > > > > > > > > > > > take 2 more months, I would > > estimate, > > > >> > before > > > >> > > it > > > >> > > > > is > > > >> > > > > > > > fully > > > >> > > > > > > > > > > > > available. > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > If we want to be on the safe side > > with > > > >> the > > > >> > > > > > > > contribution, > > > >> > > > > > > > > we > > > >> > > > > > > > > > > > > should > > > >> > > > > > > > > > > > > > > > > > contribute the source sooner and > > > adjust > > > >> it > > > >> > > > later. > > > >> > > > > > > That > > > >> > > > > > > > > > would > > > >> > > > > > > > > > > > also > > > >> > > > > > > > > > > > > > > help > > > >> > > > > > > > > > > > > > > > us > > > >> > > > > > > > > > > > > > > > > > in case things get crazy towards > the > > > 1.10 > > > >> > > > feature > > > >> > > > > > > > freeze > > > >> > > > > > > > > > and > > > >> > > > > > > > > > > it > > > >> > > > > > > > > > > > > would > > > >> > > > > > > > > > > > > > > > be > > > >> > > > > > > > > > > > > > > > > > hard to find time to review the > new > > > >> > changes. > > > >> > > > > > > > > > > > > > > > > > What would be the overhead of > > > >> contributing > > > >> > > now? > > > >> > > > > > Given > > > >> > > > > > > > > that > > > >> > > > > > > > > > > the > > > >> > > > > > > > > > > > > code > > > >> > > > > > > > > > > > > > > is > > > >> > > > > > > > > > > > > > > > > > already there, it looks like it > > would > > > be > > > >> > only > > > >> > > > > > review > > > >> > > > > > > > > > > capacity, > > > >> > > > > > > > > > > > > right? > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Best, > > > >> > > > > > > > > > > > > > > > > > Stephan > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM > > Yijie > > > >> > Shen < > > > >> > > > > > > > > > > > > > > henry.yijies...@gmail.com > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > Hi everyone! > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > Thanks for your attention and > the > > > >> > promotion > > > >> > > > of > > > >> > > > > > this > > > >> > > > > > > > > work. > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > We will prepare a FLIP as soon > as > > > >> > possible > > > >> > > > for > > > >> > > > > > more > > > >> > > > > > > > > > > specific > > > >> > > > > > > > > > > > > > > > > discussions. > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > For FLIP-27, it seems that we > have > > > not > > > >> > > > reached > > > >> > > > > a > > > >> > > > > > > > > > consensus. > > > >> > > > > > > > > > > > > > > > Therefore, > > > >> > > > > > > > > > > > > > > > > > > I will explain all the > > > functionalities > > > >> of > > > >> > > the > > > >> > > > > > > > existing > > > >> > > > > > > > > > > > > connector in > > > >> > > > > > > > > > > > > > > > > > > the FLIP (including Source, > Sink, > > > and > > > >> > > > Catalog) > > > >> > > > > to > > > >> > > > > > > > > > continue > > > >> > > > > > > > > > > > our > > > >> > > > > > > > > > > > > > > > > > > discussions in FLIP. > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > Thanks for your kind help. > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > Best, > > > >> > > > > > > > > > > > > > > > > > > Yijie > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM > > > Becket > > > >> > Qin > > > >> > > < > > > >> > > > > > > > > > > > > becket....@gmail.com> > > > >> > > > > > > > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Hi Sijie, > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > If we agree that the goal is > to > > > have > > > >> > > Pulsar > > > >> > > > > > > > connector > > > >> > > > > > > > > > in > > > >> > > > > > > > > > > > > 1.10, > > > >> > > > > > > > > > > > > > > how > > > >> > > > > > > > > > > > > > > > > > about > > > >> > > > > > > > > > > > > > > > > > > we > > > >> > > > > > > > > > > > > > > > > > > > do the following: > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar > > > >> connector > > > >> > > to > > > >> > > > > > Flink > > > >> > > > > > > > main > > > >> > > > > > > > > > > repo > > > >> > > > > > > > > > > > > as it > > > >> > > > > > > > > > > > > > > > is a > > > >> > > > > > > > > > > > > > > > > > new > > > >> > > > > > > > > > > > > > > > > > > > public interface to Flink main > > > repo. > > > >> > > > > > > > > > > > > > > > > > > > 1. Start to review the Pulsar > > sink > > > >> > right > > > >> > > > away > > > >> > > > > > as > > > >> > > > > > > > > there > > > >> > > > > > > > > > is > > > >> > > > > > > > > > > > no > > > >> > > > > > > > > > > > > > > change > > > >> > > > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > sink interface so far. > > > >> > > > > > > > > > > > > > > > > > > > 2. Wait a little bit on > FLIP-27. > > > >> Flink > > > >> > > 1.10 > > > >> > > > > is > > > >> > > > > > > > going > > > >> > > > > > > > > to > > > >> > > > > > > > > > > be > > > >> > > > > > > > > > > > > code > > > >> > > > > > > > > > > > > > > > > freeze > > > >> > > > > > > > > > > > > > > > > > in > > > >> > > > > > > > > > > > > > > > > > > > late Nov and let's say we > give a > > > >> month > > > >> > to > > > >> > > > the > > > >> > > > > > > > > > development > > > >> > > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > > review > > > >> > > > > > > > > > > > > > > > > of > > > >> > > > > > > > > > > > > > > > > > > > Pulsar connector, we need to > > have > > > >> > FLIP-27 > > > >> > > > by > > > >> > > > > > late > > > >> > > > > > > > > Oct. > > > >> > > > > > > > > > > > There > > > >> > > > > > > > > > > > > are > > > >> > > > > > > > > > > > > > > > > still > > > >> > > > > > > > > > > > > > > > > > 7 > > > >> > > > > > > > > > > > > > > > > > > > weeks. Personally I think it > is > > > >> doable. > > > >> > > If > > > >> > > > > > > FLIP-27 > > > >> > > > > > > > is > > > >> > > > > > > > > > not > > > >> > > > > > > > > > > > > ready > > > >> > > > > > > > > > > > > > > by > > > >> > > > > > > > > > > > > > > > > late > > > >> > > > > > > > > > > > > > > > > > > > Oct, we can review and check > in > > > >> Pulsar > > > >> > > > > > connector > > > >> > > > > > > > with > > > >> > > > > > > > > > the > > > >> > > > > > > > > > > > > > > existing > > > >> > > > > > > > > > > > > > > > > > source > > > >> > > > > > > > > > > > > > > > > > > > interface. This means we will > > have > > > >> > Pulsar > > > >> > > > > > > connector > > > >> > > > > > > > > in > > > >> > > > > > > > > > > > Flink > > > >> > > > > > > > > > > > > > > 1.10, > > > >> > > > > > > > > > > > > > > > > > either > > > >> > > > > > > > > > > > > > > > > > > > with or without FLIP-27. > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Because we are going to have > > > Pulsar > > > >> > sink > > > >> > > > and > > > >> > > > > > > source > > > >> > > > > > > > > > > checked > > > >> > > > > > > > > > > > > in > > > >> > > > > > > > > > > > > > > > > > > separately, > > > >> > > > > > > > > > > > > > > > > > > > it might make sense to have > two > > > >> FLIPs, > > > >> > > one > > > >> > > > > for > > > >> > > > > > > > Pulsar > > > >> > > > > > > > > > > sink > > > >> > > > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > > > another > > > >> > > > > > > > > > > > > > > > > > > for > > > >> > > > > > > > > > > > > > > > > > > > Pulsar source. And we can > start > > > the > > > >> > work > > > >> > > on > > > >> > > > > > > Pulsar > > > >> > > > > > > > > sink > > > >> > > > > > > > > > > > right > > > >> > > > > > > > > > > > > > > away. > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Thanks, > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM > > > Sijie > > > >> > Guo > > > >> > > < > > > >> > > > > > > > > > > > guosi...@gmail.com > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > Thank you Bowen and Becket. > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > What's the take from Flink > > > >> community? > > > >> > > > Shall > > > >> > > > > > we > > > >> > > > > > > > wait > > > >> > > > > > > > > > for > > > >> > > > > > > > > > > > > FLIP-27 > > > >> > > > > > > > > > > > > > > > or > > > >> > > > > > > > > > > > > > > > > > > shall we > > > >> > > > > > > > > > > > > > > > > > > > > proceed to next steps? And > > what > > > the > > > >> > > next > > > >> > > > > > steps > > > >> > > > > > > > are? > > > >> > > > > > > > > > :-) > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > Thanks, > > > >> > > > > > > > > > > > > > > > > > > > > Sijie > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 > PM > > > >> Bowen > > > >> > > Li < > > > >> > > > > > > > > > > > > bowenl...@gmail.com> > > > >> > > > > > > > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Hi, > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > I think having a Pulsar > > > connector > > > >> > in > > > >> > > > > Flink > > > >> > > > > > > can > > > >> > > > > > > > > be a > > > >> > > > > > > > > > > > good > > > >> > > > > > > > > > > > > > > mutual > > > >> > > > > > > > > > > > > > > > > > > benefit > > > >> > > > > > > > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > > > > > both communities. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Another perspective is > that > > > >> Pulsar > > > >> > > > > > connector > > > >> > > > > > > is > > > >> > > > > > > > > the > > > >> > > > > > > > > > > 1st > > > >> > > > > > > > > > > > > > > > streaming > > > >> > > > > > > > > > > > > > > > > > > > > connector > > > >> > > > > > > > > > > > > > > > > > > > > > that integrates with > Flink's > > > >> > metadata > > > >> > > > > > > > management > > > >> > > > > > > > > > > system > > > >> > > > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > > > Catalog > > > >> > > > > > > > > > > > > > > > > > > APIs. > > > >> > > > > > > > > > > > > > > > > > > > > > It'll be cool to see how > the > > > >> > > > integration > > > >> > > > > > > turns > > > >> > > > > > > > > out > > > >> > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > whether > > > >> > > > > > > > > > > > > > > > we > > > >> > > > > > > > > > > > > > > > > > > need to > > > >> > > > > > > > > > > > > > > > > > > > > > improve Flink Catalog > stack, > > > >> which > > > >> > > are > > > >> > > > > > > > currently > > > >> > > > > > > > > in > > > >> > > > > > > > > > > > > Beta, to > > > >> > > > > > > > > > > > > > > > > cater > > > >> > > > > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > > > > > streaming source/sink. > Thus > > > I'm > > > >> in > > > >> > > > favor > > > >> > > > > of > > > >> > > > > > > > > merging > > > >> > > > > > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > > > > > > connector > > > >> > > > > > > > > > > > > > > > > > > into > > > >> > > > > > > > > > > > > > > > > > > > > > Flink 1.10. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > I'd suggest to submit > > smaller > > > >> sized > > > >> > > > PRs, > > > >> > > > > > e.g. > > > >> > > > > > > > > maybe > > > >> > > > > > > > > > > one > > > >> > > > > > > > > > > > > for > > > >> > > > > > > > > > > > > > > > basic > > > >> > > > > > > > > > > > > > > > > > > > > > source/sink > functionalities > > > and > > > >> > > another > > > >> > > > > for > > > >> > > > > > > > > schema > > > >> > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > catalog > > > >> > > > > > > > > > > > > > > > > > > > > integration, > > > >> > > > > > > > > > > > > > > > > > > > > > just to make them easier > to > > > >> review. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > It doesn't seem to hurt to > > > wait > > > >> for > > > >> > > > > > FLIP-27. > > > >> > > > > > > > But > > > >> > > > > > > > > I > > > >> > > > > > > > > > > > don't > > > >> > > > > > > > > > > > > > > think > > > >> > > > > > > > > > > > > > > > > > > FLIP-27 > > > >> > > > > > > > > > > > > > > > > > > > > > should be a blocker in > cases > > > >> where > > > >> > it > > > >> > > > > > cannot > > > >> > > > > > > > make > > > >> > > > > > > > > > its > > > >> > > > > > > > > > > > way > > > >> > > > > > > > > > > > > > > into > > > >> > > > > > > > > > > > > > > > > 1.10 > > > >> > > > > > > > > > > > > > > > > > > or > > > >> > > > > > > > > > > > > > > > > > > > > > doesn't leave reasonable > > > amount > > > >> of > > > >> > > time > > > >> > > > > for > > > >> > > > > > > > > > > committers > > > >> > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > review > > > >> > > > > > > > > > > > > > > > > or > > > >> > > > > > > > > > > > > > > > > > > for > > > >> > > > > > > > > > > > > > > > > > > > > > Pulsar connector to fully > > > adapt > > > >> to > > > >> > > new > > > >> > > > > > > > > interfaces. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Bowen > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at > 3:21 > > AM > > > >> > Becket > > > >> > > > > Qin < > > > >> > > > > > > > > > > > > > > > becket....@gmail.com> > > > >> > > > > > > > > > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Hi Till, > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > You are right. It all > > > depends > > > >> on > > > >> > > when > > > >> > > > > the > > > >> > > > > > > new > > > >> > > > > > > > > > > source > > > >> > > > > > > > > > > > > > > > interface > > > >> > > > > > > > > > > > > > > > > is > > > >> > > > > > > > > > > > > > > > > > > going > > > >> > > > > > > > > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > > > > > > be ready. Personally I > > > think it > > > >> > > would > > > >> > > > > be > > > >> > > > > > > > there > > > >> > > > > > > > > in > > > >> > > > > > > > > > > > > about a > > > >> > > > > > > > > > > > > > > > month > > > >> > > > > > > > > > > > > > > > > > or > > > >> > > > > > > > > > > > > > > > > > > so. > > > >> > > > > > > > > > > > > > > > > > > > > > But > > > >> > > > > > > > > > > > > > > > > > > > > > > I could be too > optimistic. > > > It > > > >> > would > > > >> > > > > also > > > >> > > > > > be > > > >> > > > > > > > > good > > > >> > > > > > > > > > to > > > >> > > > > > > > > > > > > hear > > > >> > > > > > > > > > > > > > > what > > > >> > > > > > > > > > > > > > > > > do > > > >> > > > > > > > > > > > > > > > > > > > > Aljoscha > > > >> > > > > > > > > > > > > > > > > > > > > > > and Stephan think as > they > > > are > > > >> > also > > > >> > > > > > involved > > > >> > > > > > > > in > > > >> > > > > > > > > > > > FLIP-27. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > In general I think we > > should > > > >> have > > > >> > > > > Pulsar > > > >> > > > > > > > > > connector > > > >> > > > > > > > > > > in > > > >> > > > > > > > > > > > > Flink > > > >> > > > > > > > > > > > > > > > > 1.10, > > > >> > > > > > > > > > > > > > > > > > > > > > > preferably with the new > > > source > > > >> > > > > interface. > > > >> > > > > > > We > > > >> > > > > > > > > can > > > >> > > > > > > > > > > also > > > >> > > > > > > > > > > > > check > > > >> > > > > > > > > > > > > > > > it > > > >> > > > > > > > > > > > > > > > > in > > > >> > > > > > > > > > > > > > > > > > > right > > > >> > > > > > > > > > > > > > > > > > > > > > now > > > >> > > > > > > > > > > > > > > > > > > > > > > with old source > interface, > > > but > > > >> I > > > >> > > > > suspect > > > >> > > > > > > few > > > >> > > > > > > > > > users > > > >> > > > > > > > > > > > > will use > > > >> > > > > > > > > > > > > > > > it > > > >> > > > > > > > > > > > > > > > > > > before > > > >> > > > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > next official release. > > > >> Therefore, > > > >> > > it > > > >> > > > > > seems > > > >> > > > > > > > > > > reasonable > > > >> > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > wait a > > > >> > > > > > > > > > > > > > > > > > > little > > > >> > > > > > > > > > > > > > > > > > > > > > bit > > > >> > > > > > > > > > > > > > > > > > > > > > > to see whether we can > jump > > > to > > > >> the > > > >> > > new > > > >> > > > > > > source > > > >> > > > > > > > > > > > > interface. As > > > >> > > > > > > > > > > > > > > > long > > > >> > > > > > > > > > > > > > > > > > as > > > >> > > > > > > > > > > > > > > > > > > we > > > >> > > > > > > > > > > > > > > > > > > > > > make > > > >> > > > > > > > > > > > > > > > > > > > > > > sure Flink 1.10 has it, > > > >> waiting a > > > >> > > > > little > > > >> > > > > > > bit > > > >> > > > > > > > > > > doesn't > > > >> > > > > > > > > > > > > seem > > > >> > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > hurt > > > >> > > > > > > > > > > > > > > > > > > much. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at > > 3:59 > > > PM > > > >> > Till > > > >> > > > > > > Rohrmann > > > >> > > > > > > > < > > > >> > > > > > > > > > > > > > > > > > trohrm...@apache.org > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > I'm wondering what the > > > >> problem > > > >> > > > would > > > >> > > > > be > > > >> > > > > > > if > > > >> > > > > > > > we > > > >> > > > > > > > > > > > > committed > > > >> > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > > > > > > > > > > > > connector before the > new > > > >> source > > > >> > > > > > interface > > > >> > > > > > > > is > > > >> > > > > > > > > > > ready. > > > >> > > > > > > > > > > > > If I > > > >> > > > > > > > > > > > > > > > > > > understood > > > >> > > > > > > > > > > > > > > > > > > > > it > > > >> > > > > > > > > > > > > > > > > > > > > > > > correctly, then we > need > > to > > > >> > > support > > > >> > > > > the > > > >> > > > > > > old > > > >> > > > > > > > > > source > > > >> > > > > > > > > > > > > > > interface > > > >> > > > > > > > > > > > > > > > > > > anyway > > > >> > > > > > > > > > > > > > > > > > > > > for > > > >> > > > > > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > > existing connectors. > By > > > >> > checking > > > >> > > it > > > >> > > > > in > > > >> > > > > > > > early > > > >> > > > > > > > > I > > > >> > > > > > > > > > > > could > > > >> > > > > > > > > > > > > see > > > >> > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > benefit > > > >> > > > > > > > > > > > > > > > > > > > > > that > > > >> > > > > > > > > > > > > > > > > > > > > > > > our users could start > > > using > > > >> the > > > >> > > > > > connector > > > >> > > > > > > > > > > earlier. > > > >> > > > > > > > > > > > > > > > Moreover, > > > >> > > > > > > > > > > > > > > > > it > > > >> > > > > > > > > > > > > > > > > > > would > > > >> > > > > > > > > > > > > > > > > > > > > > > > prevent that the > Pulsar > > > >> > > integration > > > >> > > > > is > > > >> > > > > > > > being > > > >> > > > > > > > > > > > delayed > > > >> > > > > > > > > > > > > in > > > >> > > > > > > > > > > > > > > > case > > > >> > > > > > > > > > > > > > > > > > > that the > > > >> > > > > > > > > > > > > > > > > > > > > > > > source interface > should > > be > > > >> > > delayed. > > > >> > > > > The > > > >> > > > > > > > only > > > >> > > > > > > > > > > > > downside I > > > >> > > > > > > > > > > > > > > see > > > >> > > > > > > > > > > > > > > > > is > > > >> > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > extra > > > >> > > > > > > > > > > > > > > > > > > > > > > > review effort and > > > potential > > > >> > fixes > > > >> > > > > which > > > >> > > > > > > > might > > > >> > > > > > > > > > be > > > >> > > > > > > > > > > > > > > irrelevant > > > >> > > > > > > > > > > > > > > > > for > > > >> > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > new > > > >> > > > > > > > > > > > > > > > > > > > > > > > source interface > > > >> > implementation. > > > >> > > I > > > >> > > > > > guess > > > >> > > > > > > it > > > >> > > > > > > > > > > mainly > > > >> > > > > > > > > > > > > > > depends > > > >> > > > > > > > > > > > > > > > on > > > >> > > > > > > > > > > > > > > > > > how > > > >> > > > > > > > > > > > > > > > > > > > > > certain > > > >> > > > > > > > > > > > > > > > > > > > > > > > we are when the new > > source > > > >> > > > interface > > > >> > > > > > will > > > >> > > > > > > > be > > > >> > > > > > > > > > > ready. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > >> > > > > > > > > > > > > > > > > > > > > > > > Till > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at > > > 8:56 > > > >> AM > > > >> > > > Becket > > > >> > > > > > > Qin < > > > >> > > > > > > > > > > > > > > > > > becket....@gmail.com> > > > >> > > > > > > > > > > > > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > Hi Sijie and Yijie, > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > Thanks for sharing > > your > > > >> > > thoughts. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > Just want to have > some > > > >> update > > > >> > > on > > > >> > > > > > > FLIP-27. > > > >> > > > > > > > > > > > Although > > > >> > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > FLIP > > > >> > > > > > > > > > > > > > > > > > > wiki > > > >> > > > > > > > > > > > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > > > > > > > > > > > discussion thread > has > > > been > > > >> > > quiet > > > >> > > > > for > > > >> > > > > > > some > > > >> > > > > > > > > > > time, a > > > >> > > > > > > > > > > > > few > > > >> > > > > > > > > > > > > > > > > > > committer / > > > >> > > > > > > > > > > > > > > > > > > > > > > > > contributors in > Flink > > > >> > community > > > >> > > > > were > > > >> > > > > > > > > actually > > > >> > > > > > > > > > > > > > > prototyping > > > >> > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > entire > > > >> > > > > > > > > > > > > > > > > > > > > > > > thing. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > We have made some > good > > > >> > progress > > > >> > > > > there > > > >> > > > > > > but > > > >> > > > > > > > > > want > > > >> > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > update > > > >> > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > FLIP > > > >> > > > > > > > > > > > > > > > > > > > > > wiki > > > >> > > > > > > > > > > > > > > > > > > > > > > > > after the entire > thing > > > is > > > >> > > > verified > > > >> > > > > to > > > >> > > > > > > > work > > > >> > > > > > > > > in > > > >> > > > > > > > > > > > case > > > >> > > > > > > > > > > > > > > there > > > >> > > > > > > > > > > > > > > > > are > > > >> > > > > > > > > > > > > > > > > > > some > > > >> > > > > > > > > > > > > > > > > > > > > > last > > > >> > > > > > > > > > > > > > > > > > > > > > > > > minute surprise in > the > > > >> > > > > > implementation. > > > >> > > > > > > I > > > >> > > > > > > > > > don't > > > >> > > > > > > > > > > > > have an > > > >> > > > > > > > > > > > > > > > > exact > > > >> > > > > > > > > > > > > > > > > > > ETA > > > >> > > > > > > > > > > > > > > > > > > > > yet, > > > >> > > > > > > > > > > > > > > > > > > > > > > > but I > > > >> > > > > > > > > > > > > > > > > > > > > > > > > guess it is going to > > be > > > >> > within > > > >> > > a > > > >> > > > > > month > > > >> > > > > > > or > > > >> > > > > > > > > so. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > I am happy to review > > the > > > >> > > current > > > >> > > > > > Flink > > > >> > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > connector > > > >> > > > > > > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > > > > > see if > > > >> > > > > > > > > > > > > > > > > > > > > it > > > >> > > > > > > > > > > > > > > > > > > > > > > > would > > > >> > > > > > > > > > > > > > > > > > > > > > > > > fit in FLIP-27. It > > > would be > > > >> > > good > > > >> > > > to > > > >> > > > > > > avoid > > > >> > > > > > > > > the > > > >> > > > > > > > > > > > case > > > >> > > > > > > > > > > > > that > > > >> > > > > > > > > > > > > > > > we > > > >> > > > > > > > > > > > > > > > > > > checked > > > >> > > > > > > > > > > > > > > > > > > > > in > > > >> > > > > > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector > with > > > some > > > >> > > review > > > >> > > > > > > efforts > > > >> > > > > > > > > and > > > >> > > > > > > > > > > > > shortly > > > >> > > > > > > > > > > > > > > > after > > > >> > > > > > > > > > > > > > > > > > > that > > > >> > > > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > new > > > >> > > > > > > > > > > > > > > > > > > > > > > > > Source interface is > > > ready. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) > Qin > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 > at > > > 8:39 > > > >> > AM > > > >> > > > > Yijie > > > >> > > > > > > > Shen < > > > >> > > > > > > > > > > > > > > > > > > > > henry.yijies...@gmail.com > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for all the > > > >> feedback > > > >> > > and > > > >> > > > > > > > > > suggestions! > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > As Sijie said, the > > > goal > > > >> of > > > >> > > the > > > >> > > > > > > > connector > > > >> > > > > > > > > > has > > > >> > > > > > > > > > > > > always > > > >> > > > > > > > > > > > > > > > been > > > >> > > > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > > > > provide > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > users with the > > latest > > > >> > > features > > > >> > > > of > > > >> > > > > > > both > > > >> > > > > > > > > > > systems > > > >> > > > > > > > > > > > as > > > >> > > > > > > > > > > > > > > soon > > > >> > > > > > > > > > > > > > > > as > > > >> > > > > > > > > > > > > > > > > > > > > possible. > > > >> > > > > > > > > > > > > > > > > > > > > > > We > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > propose to > > contribute > > > the > > > >> > > > > connector > > > >> > > > > > > to > > > >> > > > > > > > > > Flink > > > >> > > > > > > > > > > > and > > > >> > > > > > > > > > > > > hope > > > >> > > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > get > > > >> > > > > > > > > > > > > > > > > > > more > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > suggestions and > > > feedback > > > >> > from > > > >> > > > > Flink > > > >> > > > > > > > > experts > > > >> > > > > > > > > > > to > > > >> > > > > > > > > > > > > ensure > > > >> > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > high > > > >> > > > > > > > > > > > > > > > > > > > > > > quality > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > of the connector. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > For FLIP-27, we > > > noticed > > > >> its > > > >> > > > > > existence > > > >> > > > > > > > at > > > >> > > > > > > > > > the > > > >> > > > > > > > > > > > > > > beginning > > > >> > > > > > > > > > > > > > > > of > > > >> > > > > > > > > > > > > > > > > > > > > reworking > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > the connector > > > >> > implementation > > > >> > > > > based > > > >> > > > > > on > > > >> > > > > > > > > Flink > > > >> > > > > > > > > > > > 1.9; > > > >> > > > > > > > > > > > > we > > > >> > > > > > > > > > > > > > > > also > > > >> > > > > > > > > > > > > > > > > > > wanted > > > >> > > > > > > > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > build a connector > > that > > > >> > > supports > > > >> > > > > > both > > > >> > > > > > > > > batch > > > >> > > > > > > > > > > and > > > >> > > > > > > > > > > > > stream > > > >> > > > > > > > > > > > > > > > > > > computing > > > >> > > > > > > > > > > > > > > > > > > > > > based > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > on it. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > However, it has > been > > > >> > inactive > > > >> > > > for > > > >> > > > > > > some > > > >> > > > > > > > > > time, > > > >> > > > > > > > > > > so > > > >> > > > > > > > > > > > > we > > > >> > > > > > > > > > > > > > > > > decided > > > >> > > > > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > > > > > provide > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > a connector with > > most > > > of > > > >> > the > > > >> > > > new > > > >> > > > > > > > > features, > > > >> > > > > > > > > > > such > > > >> > > > > > > > > > > > > as > > > >> > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > new > > > >> > > > > > > > > > > > > > > > > > > type > > > >> > > > > > > > > > > > > > > > > > > > > > > system > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > and the new > catalog > > > API > > > >> > > first. > > > >> > > > We > > > >> > > > > > > will > > > >> > > > > > > > > pay > > > >> > > > > > > > > > > > > attention > > > >> > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > progress > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > of FLIP-27 > > continually > > > >> and > > > >> > > > > > > incorporate > > > >> > > > > > > > it > > > >> > > > > > > > > > > with > > > >> > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > connector > > > >> > > > > > > > > > > > > > > > > > > as > > > >> > > > > > > > > > > > > > > > > > > > > > soon > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > as possible. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Regarding the test > > > status > > > >> > of > > > >> > > > the > > > >> > > > > > > > > connector, > > > >> > > > > > > > > > > we > > > >> > > > > > > > > > > > > are > > > >> > > > > > > > > > > > > > > > > > following > > > >> > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > other > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > connectors' test > in > > > Flink > > > >> > > > > > repository > > > >> > > > > > > > and > > > >> > > > > > > > > > > aimed > > > >> > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > provide > > > >> > > > > > > > > > > > > > > > > > > > > > throughout > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > tests as we could. > > We > > > are > > > >> > > also > > > >> > > > > > happy > > > >> > > > > > > to > > > >> > > > > > > > > > hear > > > >> > > > > > > > > > > > > > > > suggestions > > > >> > > > > > > > > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > supervision from > the > > > >> Flink > > > >> > > > > > community > > > >> > > > > > > to > > > >> > > > > > > > > > > improve > > > >> > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > stability and > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > performance of the > > > >> > connector > > > >> > > > > > > > > continuously. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Yijie > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, > 2019 > > at > > > >> 5:59 > > > >> > > AM > > > >> > > > > > Sijie > > > >> > > > > > > > Guo > > > >> > > > > > > > > < > > > >> > > > > > > > > > > > > > > > > > guosi...@gmail.com > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks everyone > > for > > > the > > > >> > > > > comments > > > >> > > > > > > and > > > >> > > > > > > > > > > > feedback. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > It seems to me > > that > > > the > > > >> > > main > > > >> > > > > > > question > > > >> > > > > > > > > > here > > > >> > > > > > > > > > > is > > > >> > > > > > > > > > > > > > > about - > > > >> > > > > > > > > > > > > > > > > > "how > > > >> > > > > > > > > > > > > > > > > > > can > > > >> > > > > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > community > maintain > > > the > > > >> > > > > > connector?". > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Here are two > > > thoughts > > > >> > from > > > >> > > > > > myself. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > 1) I think how > and > > > >> where > > > >> > to > > > >> > > > > host > > > >> > > > > > > this > > > >> > > > > > > > > > > > > integration > > > >> > > > > > > > > > > > > > > is > > > >> > > > > > > > > > > > > > > > > kind > > > >> > > > > > > > > > > > > > > > > > > of > > > >> > > > > > > > > > > > > > > > > > > > > less > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > important > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > here. I believe > > > there > > > >> can > > > >> > > be > > > >> > > > > many > > > >> > > > > > > > ways > > > >> > > > > > > > > to > > > >> > > > > > > > > > > > > achieve > > > >> > > > > > > > > > > > > > > it. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > As part of the > > > >> > > contribution, > > > >> > > > > what > > > >> > > > > > > we > > > >> > > > > > > > > are > > > >> > > > > > > > > > > > > looking > > > >> > > > > > > > > > > > > > > for > > > >> > > > > > > > > > > > > > > > > here > > > >> > > > > > > > > > > > > > > > > > > is > > > >> > > > > > > > > > > > > > > > > > > > > how > > > >> > > > > > > > > > > > > > > > > > > > > > > > these > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > two > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > communities can > > > build > > > >> the > > > >> > > > > > > > collaboration > > > >> > > > > > > > > > > > > > > relationship > > > >> > > > > > > > > > > > > > > > on > > > >> > > > > > > > > > > > > > > > > > > > > > developing > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > the integration > > > between > > > >> > > > Pulsar > > > >> > > > > > and > > > >> > > > > > > > > Flink. > > > >> > > > > > > > > > > > Even > > > >> > > > > > > > > > > > > we > > > >> > > > > > > > > > > > > > > can > > > >> > > > > > > > > > > > > > > > > try > > > >> > > > > > > > > > > > > > > > > > > our > > > >> > > > > > > > > > > > > > > > > > > > > > best > > > >> > > > > > > > > > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > catch > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > up all the > updates > > > in > > > >> > Flink > > > >> > > > > > > > community. > > > >> > > > > > > > > We > > > >> > > > > > > > > > > are > > > >> > > > > > > > > > > > > still > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > facing the fact > > > that we > > > >> > > have > > > >> > > > > less > > > >> > > > > > > > > > > experiences > > > >> > > > > > > > > > > > > in > > > >> > > > > > > > > > > > > > > > Flink > > > >> > > > > > > > > > > > > > > > > > than > > > >> > > > > > > > > > > > > > > > > > > > > folks > > > >> > > > > > > > > > > > > > > > > > > > > > > in > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > community. In > > order > > > to > > > >> > make > > > >> > > > > sure > > > >> > > > > > we > > > >> > > > > > > > > > > maintain > > > >> > > > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > > > deliver > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > a high-quality > > > >> > pulsar-flink > > > >> > > > > > > > integration > > > >> > > > > > > > > > to > > > >> > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > users > > > >> > > > > > > > > > > > > > > > > who > > > >> > > > > > > > > > > > > > > > > > > use > > > >> > > > > > > > > > > > > > > > > > > > > both > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > technologies, we > > > need > > > >> > some > > > >> > > > help > > > >> > > > > > > from > > > >> > > > > > > > > the > > > >> > > > > > > > > > > > > experts > > > >> > > > > > > > > > > > > > > from > > > >> > > > > > > > > > > > > > > > > > Flink > > > >> > > > > > > > > > > > > > > > > > > > > > > > community. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > 2) We have been > > > >> following > > > >> > > > > FLIP-27 > > > >> > > > > > > > for a > > > >> > > > > > > > > > > > while. > > > >> > > > > > > > > > > > > > > > > Originally > > > >> > > > > > > > > > > > > > > > > > > we > > > >> > > > > > > > > > > > > > > > > > > > > were > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > thinking > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > of contributing > > the > > > >> > > > connectors > > > >> > > > > > back > > > >> > > > > > > > > after > > > >> > > > > > > > > > > > > > > integrating > > > >> > > > > > > > > > > > > > > > > > with > > > >> > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > new API > introduced > > > in > > > >> > > > FLIP-27. > > > >> > > > > > But > > > >> > > > > > > we > > > >> > > > > > > > > > > decided > > > >> > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > initiate > > > >> > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > conversation > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > as early as > > > possible. > > > >> > > Because > > > >> > > > > we > > > >> > > > > > > > > believe > > > >> > > > > > > > > > > > there > > > >> > > > > > > > > > > > > are > > > >> > > > > > > > > > > > > > > > more > > > >> > > > > > > > > > > > > > > > > > > > > benefits > > > >> > > > > > > > > > > > > > > > > > > > > > > > doing > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > it now rather > than > > > >> later. > > > >> > > As > > > >> > > > > part > > > >> > > > > > > of > > > >> > > > > > > > > > > > > contribution, > > > >> > > > > > > > > > > > > > > it > > > >> > > > > > > > > > > > > > > > > can > > > >> > > > > > > > > > > > > > > > > > > help > > > >> > > > > > > > > > > > > > > > > > > > > > > Flink > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > community > > understand > > > >> more > > > >> > > > about > > > >> > > > > > > > Pulsar > > > >> > > > > > > > > > and > > > >> > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > potential > > > >> > > > > > > > > > > > > > > > > > > > > > > integration > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > points. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Also we can also > > > help > > > >> > Flink > > > >> > > > > > > community > > > >> > > > > > > > > > > verify > > > >> > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > new > > > >> > > > > > > > > > > > > > > > > > > connector > > > >> > > > > > > > > > > > > > > > > > > > > > API > > > >> > > > > > > > > > > > > > > > > > > > > > > as > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > well > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > as other new API > > > (e.g. > > > >> > > > catalog > > > >> > > > > > > API). > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Sijie > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, > > 2019 > > > at > > > >> > 5:24 > > > >> > > > AM > > > >> > > > > > > Becket > > > >> > > > > > > > > > Qin < > > > >> > > > > > > > > > > > > > > > > > > > > becket....@gmail.com> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yijie, > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the > > > >> interest > > > >> > > in > > > >> > > > > > > > > contributing > > > >> > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > > > > > > > connector. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > In general, I > > > think > > > >> > > having > > > >> > > > > > Pulsar > > > >> > > > > > > > > > > connector > > > >> > > > > > > > > > > > > with > > > >> > > > > > > > > > > > > > > > > strong > > > >> > > > > > > > > > > > > > > > > > > > > support > > > >> > > > > > > > > > > > > > > > > > > > > > > is > > > >> > > > > > > > > > > > > > > > > > > > > > > > a > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > valuable > > addition > > > to > > > >> > > Flink. > > > >> > > > > So > > > >> > > > > > I > > > >> > > > > > > am > > > >> > > > > > > > > > happy > > > >> > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > shepherd > > > >> > > > > > > > > > > > > > > > > > > this > > > >> > > > > > > > > > > > > > > > > > > > > > > effort. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Meanwhile, I > > would > > > >> also > > > >> > > > like > > > >> > > > > to > > > >> > > > > > > > > provide > > > >> > > > > > > > > > > > some > > > >> > > > > > > > > > > > > > > > context > > > >> > > > > > > > > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > > > > > > > recent > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > efforts on > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > the Flink > > > connectors > > > >> > > > > ecosystem. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > The current > way > > > Flink > > > >> > > > > maintains > > > >> > > > > > > its > > > >> > > > > > > > > > > > > connector has > > > >> > > > > > > > > > > > > > > > hit > > > >> > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > > scalability > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > bar. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > With more and > > more > > > >> > > > connectors > > > >> > > > > > > > coming > > > >> > > > > > > > > > into > > > >> > > > > > > > > > > > > Flink > > > >> > > > > > > > > > > > > > > > repo, > > > >> > > > > > > > > > > > > > > > > > we > > > >> > > > > > > > > > > > > > > > > > > are > > > >> > > > > > > > > > > > > > > > > > > > > > > > facing a > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > few > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > problems such > as > > > long > > > >> > > build > > > >> > > > > and > > > >> > > > > > > > > testing > > > >> > > > > > > > > > > > > time. To > > > >> > > > > > > > > > > > > > > > > > address > > > >> > > > > > > > > > > > > > > > > > > this > > > >> > > > > > > > > > > > > > > > > > > > > > > > > problem, > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > we > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > have attempted > > to > > > do > > > >> > the > > > >> > > > > > > following: > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Split out > the > > > >> > > connectors > > > >> > > > > > into > > > >> > > > > > > a > > > >> > > > > > > > > > > separate > > > >> > > > > > > > > > > > > > > > > repository. > > > >> > > > > > > > > > > > > > > > > > > This > > > >> > > > > > > > > > > > > > > > > > > > > is > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > temporarily > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > on hold due to > > > >> > potential > > > >> > > > > > solution > > > >> > > > > > > > to > > > >> > > > > > > > > > > > shorten > > > >> > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > build > > > >> > > > > > > > > > > > > > > > > > > time. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Encourage > the > > > >> > > connectors > > > >> > > > > to > > > >> > > > > > > stay > > > >> > > > > > > > > as > > > >> > > > > > > > > > > > > ecosystem > > > >> > > > > > > > > > > > > > > > > > project > > > >> > > > > > > > > > > > > > > > > > > > > while > > > >> > > > > > > > > > > > > > > > > > > > > > > > Flink > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > tries > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > to provide > good > > > >> support > > > >> > > for > > > >> > > > > > > > > > functionality > > > >> > > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > > > > > compatibility > > > >> > > > > > > > > > > > > > > > > > > > > > > tests. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Robert > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > has driven to > > > create > > > >> a > > > >> > > > Flink > > > >> > > > > > > > > Ecosystem > > > >> > > > > > > > > > > > > project > > > >> > > > > > > > > > > > > > > > > website > > > >> > > > > > > > > > > > > > > > > > > and it > > > >> > > > > > > > > > > > > > > > > > > > > > is > > > >> > > > > > > > > > > > > > > > > > > > > > > > > going > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > through some > > final > > > >> > > approval > > > >> > > > > > > > process. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Given the > above > > > >> > efforts, > > > >> > > it > > > >> > > > > > would > > > >> > > > > > > > be > > > >> > > > > > > > > > > great > > > >> > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > first > > > >> > > > > > > > > > > > > > > > > see > > > >> > > > > > > > > > > > > > > > > > > if we > > > >> > > > > > > > > > > > > > > > > > > > > > can > > > >> > > > > > > > > > > > > > > > > > > > > > > > > have > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar > connector > > > as > > > >> an > > > >> > > > > > ecosystem > > > >> > > > > > > > > > project > > > >> > > > > > > > > > > > with > > > >> > > > > > > > > > > > > > > great > > > >> > > > > > > > > > > > > > > > > > > support. > > > >> > > > > > > > > > > > > > > > > > > > > It > > > >> > > > > > > > > > > > > > > > > > > > > > > > would > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > be > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > good to hear > how > > > the > > > >> > > Flink > > > >> > > > > > Pulsar > > > >> > > > > > > > > > > connector > > > >> > > > > > > > > > > > > is > > > >> > > > > > > > > > > > > > > > tested > > > >> > > > > > > > > > > > > > > > > > > > > currently > > > >> > > > > > > > > > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > > > > > > > > see > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > if > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > we can learn > > > >> something > > > >> > to > > > >> > > > > > > maintain > > > >> > > > > > > > it > > > >> > > > > > > > > > as > > > >> > > > > > > > > > > an > > > >> > > > > > > > > > > > > > > > ecosystem > > > >> > > > > > > > > > > > > > > > > > > project > > > >> > > > > > > > > > > > > > > > > > > > > > > with > > > >> > > > > > > > > > > > > > > > > > > > > > > > > good > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > quality and > test > > > >> > > coverage. > > > >> > > > If > > > >> > > > > > the > > > >> > > > > > > > > > quality > > > >> > > > > > > > > > > > as > > > >> > > > > > > > > > > > > an > > > >> > > > > > > > > > > > > > > > > > ecosystem > > > >> > > > > > > > > > > > > > > > > > > > > > project > > > >> > > > > > > > > > > > > > > > > > > > > > > > is > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > hard > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > to guarantee, > we > > > may > > > >> as > > > >> > > > well > > > >> > > > > > > adopt > > > >> > > > > > > > it > > > >> > > > > > > > > > > into > > > >> > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > main > > > >> > > > > > > > > > > > > > > > > > repo. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > BTW, another > > > ongoing > > > >> > > effort > > > >> > > > > is > > > >> > > > > > > > > FLIP-27 > > > >> > > > > > > > > > > > where > > > >> > > > > > > > > > > > > we > > > >> > > > > > > > > > > > > > > are > > > >> > > > > > > > > > > > > > > > > > > making > > > >> > > > > > > > > > > > > > > > > > > > > > > changes > > > >> > > > > > > > > > > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink source > > > >> connector > > > >> > > > > > > architecture > > > >> > > > > > > > > and > > > >> > > > > > > > > > > > > > > interface. > > > >> > > > > > > > > > > > > > > > > This > > > >> > > > > > > > > > > > > > > > > > > > > change > > > >> > > > > > > > > > > > > > > > > > > > > > > will > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > likely > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > land in 1.10. > > > >> Therefore > > > >> > > > > timing > > > >> > > > > > > > wise, > > > >> > > > > > > > > if > > > >> > > > > > > > > > > we > > > >> > > > > > > > > > > > > are > > > >> > > > > > > > > > > > > > > > going > > > >> > > > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > > have > > > >> > > > > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > connector in > > main > > > >> > repo, I > > > >> > > > am > > > >> > > > > > > > > wondering > > > >> > > > > > > > > > if > > > >> > > > > > > > > > > > we > > > >> > > > > > > > > > > > > > > should > > > >> > > > > > > > > > > > > > > > > > hold > > > >> > > > > > > > > > > > > > > > > > > a > > > >> > > > > > > > > > > > > > > > > > > > > > little > > > >> > > > > > > > > > > > > > > > > > > > > > > > bit > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > let the Pulsar > > > >> > connector > > > >> > > > > adapt > > > >> > > > > > to > > > >> > > > > > > > the > > > >> > > > > > > > > > new > > > >> > > > > > > > > > > > > > > interface > > > >> > > > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > > avoid > > > >> > > > > > > > > > > > > > > > > > > > > > > > shortly > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > deprecated > work? > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie > > (Becket) > > > Qin > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, > > > 2019 > > > >> at > > > >> > > 4:32 > > > >> > > > > PM > > > >> > > > > > > > > Chesnay > > > >> > > > > > > > > > > > > Schepler < > > > >> > > > > > > > > > > > > > > > > > > > > > > > ches...@apache.org> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm quite > > > worried > > > >> > that > > > >> > > we > > > >> > > > > may > > > >> > > > > > > end > > > >> > > > > > > > > up > > > >> > > > > > > > > > > > > repeating > > > >> > > > > > > > > > > > > > > > > > history. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > There were > > > already > > > >> 2 > > > >> > > > > attempts > > > >> > > > > > > at > > > >> > > > > > > > > > > > > contributing a > > > >> > > > > > > > > > > > > > > > > > pulsar > > > >> > > > > > > > > > > > > > > > > > > > > > > connector, > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > both > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > of which > > failed > > > >> > because > > > >> > > > no > > > >> > > > > > > > > committer > > > >> > > > > > > > > > > was > > > >> > > > > > > > > > > > > > > getting > > > >> > > > > > > > > > > > > > > > > > > involved, > > > >> > > > > > > > > > > > > > > > > > > > > > > > despite > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > contributor > > > >> opening a > > > >> > > > > > dedicated > > > >> > > > > > > > > > > > discussion > > > >> > > > > > > > > > > > > > > thread > > > >> > > > > > > > > > > > > > > > > > > about the > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > contribution > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > beforehand > and > > > >> > getting > > > >> > > > > > several > > > >> > > > > > > > +1's > > > >> > > > > > > > > > > from > > > >> > > > > > > > > > > > > > > > > committers. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > We should > > really > > > >> make > > > >> > > > sure > > > >> > > > > > that > > > >> > > > > > > > if > > > >> > > > > > > > > we > > > >> > > > > > > > > > > > > > > > > welcome/approve > > > >> > > > > > > > > > > > > > > > > > > such > > > >> > > > > > > > > > > > > > > > > > > > > a > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > contribution > > it > > > >> will > > > >> > > > > actually > > > >> > > > > > > get > > > >> > > > > > > > > the > > > >> > > > > > > > > > > > > attention > > > >> > > > > > > > > > > > > > > > it > > > >> > > > > > > > > > > > > > > > > > > > > deserves. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > As such, I'm > > > >> inclined > > > >> > > to > > > >> > > > > > > > recommend > > > >> > > > > > > > > > > > > maintaining > > > >> > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > connector > > > >> > > > > > > > > > > > > > > > > > > > > > > > > outside > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > of > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink. We > > could > > > >> link > > > >> > to > > > >> > > > it > > > >> > > > > > from > > > >> > > > > > > > the > > > >> > > > > > > > > > > > > > > documentation > > > >> > > > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > > give > > > >> > > > > > > > > > > > > > > > > > > > > it > > > >> > > > > > > > > > > > > > > > > > > > > > > more > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > exposure. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > With the > > > upcoming > > > >> > page > > > >> > > > for > > > >> > > > > > > > sharing > > > >> > > > > > > > > > > > > artifacts > > > >> > > > > > > > > > > > > > > > among > > > >> > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > community > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > (what's > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > the state of > > > that > > > >> > > > anyway?), > > > >> > > > > > > this > > > >> > > > > > > > > may > > > >> > > > > > > > > > > be a > > > >> > > > > > > > > > > > > > > better > > > >> > > > > > > > > > > > > > > > > > > option. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > On > 04/09/2019 > > > >> 10:16, > > > >> > > Till > > > >> > > > > > > > Rohrmann > > > >> > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi > everyone, > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks a > lot > > > for > > > >> > > > starting > > > >> > > > > > > this > > > >> > > > > > > > > > > > discussion > > > >> > > > > > > > > > > > > > > > Yijie. > > > >> > > > > > > > > > > > > > > > > I > > > >> > > > > > > > > > > > > > > > > > > think > > > >> > > > > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > connector > > > would > > > >> be > > > >> > a > > > >> > > > very > > > >> > > > > > > > > valuable > > > >> > > > > > > > > > > > > addition > > > >> > > > > > > > > > > > > > > > since > > > >> > > > > > > > > > > > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > > > > > > > > > > > > becomes > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > more > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > more > popular > > > and > > > >> it > > > >> > > > would > > > >> > > > > > > > further > > > >> > > > > > > > > > > > expand > > > >> > > > > > > > > > > > > > > > Flink's > > > >> > > > > > > > > > > > > > > > > > > > > > > > > interoperability. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Also > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > from a > > project > > > >> > > > > perspective > > > >> > > > > > it > > > >> > > > > > > > > makes > > > >> > > > > > > > > > > > > sense for > > > >> > > > > > > > > > > > > > > > me > > > >> > > > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > > place > > > >> > > > > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > connector > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > in > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > the > > downstream > > > >> > > project. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My main > > > >> > > > concern/question > > > >> > > > > is > > > >> > > > > > > how > > > >> > > > > > > > > can > > > >> > > > > > > > > > > the > > > >> > > > > > > > > > > > > Flink > > > >> > > > > > > > > > > > > > > > > > > community > > > >> > > > > > > > > > > > > > > > > > > > > > > > maintain > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > connector? > > We > > > >> have > > > >> > > seen > > > >> > > > > in > > > >> > > > > > > the > > > >> > > > > > > > > past > > > >> > > > > > > > > > > > that > > > >> > > > > > > > > > > > > > > > > connectors > > > >> > > > > > > > > > > > > > > > > > > are > > > >> > > > > > > > > > > > > > > > > > > > > > some > > > >> > > > > > > > > > > > > > > > > > > > > > > of > > > >> > > > > > > > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > most > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > actively > > > >> developed > > > >> > > > > > components > > > >> > > > > > > > > > because > > > >> > > > > > > > > > > > > they > > > >> > > > > > > > > > > > > > > need > > > >> > > > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > be > > > >> > > > > > > > > > > > > > > > > > > > > kept > > > >> > > > > > > > > > > > > > > > > > > > > > in > > > >> > > > > > > > > > > > > > > > > > > > > > > > > sync > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > with > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > external > > > system > > > >> and > > > >> > > > with > > > >> > > > > > > Flink. > > > >> > > > > > > > > > Given > > > >> > > > > > > > > > > > > that > > > >> > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > > > > > > > > > > > community > > > >> > > > > > > > > > > > > > > > > > > > > > > > > is > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > willing > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > to help > with > > > >> > > > maintaining, > > > >> > > > > > > > > improving > > > >> > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > > evolving > > > >> > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > connector, > > > >> > > > > > > > > > > > > > > > > > > > > > > > > I'm > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > optimistic > > > that > > > >> we > > > >> > > can > > > >> > > > > > > achieve > > > >> > > > > > > > > > this. > > > >> > > > > > > > > > > > > Hence, > > > >> > > > > > > > > > > > > > > +1 > > > >> > > > > > > > > > > > > > > > > for > > > >> > > > > > > > > > > > > > > > > > > > > > > contributing > > > >> > > > > > > > > > > > > > > > > > > > > > > > > it > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > back > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Till > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, > Sep > > 4, > > > >> 2019 > > > >> > > at > > > >> > > > > 2:03 > > > >> > > > > > > AM > > > >> > > > > > > > > > Sijie > > > >> > > > > > > > > > > > Guo > > > >> > > > > > > > > > > > > < > > > >> > > > > > > > > > > > > > > > > > > > > > guosi...@gmail.com > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi Yun, > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Since I > was > > > the > > > >> > main > > > >> > > > > > driver > > > >> > > > > > > > > behind > > > >> > > > > > > > > > > > > > > FLINK-9641 > > > >> > > > > > > > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > > > > > > > > > FLINK-9168, > > > >> > > > > > > > > > > > > > > > > > > > > > > > > let > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > me > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > try to > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> add more > > > context > > > >> > on > > > >> > > > > this. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > FLINK-9641 > > > and > > > >> > > > > FLINK-9168 > > > >> > > > > > > was > > > >> > > > > > > > > > > created > > > >> > > > > > > > > > > > > for > > > >> > > > > > > > > > > > > > > > > bringing > > > >> > > > > > > > > > > > > > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > > > > > > > > > > as > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > source > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> sink for > > > Flink. > > > >> > The > > > >> > > > > > > > integration > > > >> > > > > > > > > > was > > > >> > > > > > > > > > > > done > > > >> > > > > > > > > > > > > > > with > > > >> > > > > > > > > > > > > > > > > > Flink > > > >> > > > > > > > > > > > > > > > > > > > > 1.6.0. > > > >> > > > > > > > > > > > > > > > > > > > > > > We > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > sent out > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > pull > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> requests > > > about a > > > >> > > year > > > >> > > > > ago > > > >> > > > > > > and > > > >> > > > > > > > we > > > >> > > > > > > > > > > ended > > > >> > > > > > > > > > > > > up > > > >> > > > > > > > > > > > > > > > > > > maintaining > > > >> > > > > > > > > > > > > > > > > > > > > > those > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > connectors > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > in > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar > for > > > >> Pulsar > > > >> > > > users > > > >> > > > > to > > > >> > > > > > > use > > > >> > > > > > > > > > Flink > > > >> > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > process > > > >> > > > > > > > > > > > > > > > > > > event > > > >> > > > > > > > > > > > > > > > > > > > > > > streams > > > >> > > > > > > > > > > > > > > > > > > > > > > > in > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> (See > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink > > > >> > > > > > > > > > > > > > > > > > > > > > > > > ). > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > The > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> 1.6 > > > integration > > > >> is > > > >> > > > > pretty > > > >> > > > > > > > simple > > > >> > > > > > > > > > and > > > >> > > > > > > > > > > > > there > > > >> > > > > > > > > > > > > > > is > > > >> > > > > > > > > > > > > > > > no > > > >> > > > > > > > > > > > > > > > > > > schema > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > considerations. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> In the > past > > > >> year, > > > >> > we > > > >> > > > > have > > > >> > > > > > > > made a > > > >> > > > > > > > > > lot > > > >> > > > > > > > > > > > of > > > >> > > > > > > > > > > > > > > > changes > > > >> > > > > > > > > > > > > > > > > in > > > >> > > > > > > > > > > > > > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > brought > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar > > > schema as > > > >> > the > > > >> > > > > > > > first-class > > > >> > > > > > > > > > > > > citizen in > > > >> > > > > > > > > > > > > > > > > > Pulsar. > > > >> > > > > > > > > > > > > > > > > > > We > > > >> > > > > > > > > > > > > > > > > > > > > > also > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > integrated > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > with > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> other > > > computing > > > >> > > > engines > > > >> > > > > > for > > > >> > > > > > > > > > > processing > > > >> > > > > > > > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > > > > > > event > > > >> > > > > > > > > > > > > > > > > > > > > > streams > > > >> > > > > > > > > > > > > > > > > > > > > > > > with > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> schema. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> It led us > > to > > > >> > rethink > > > >> > > > how > > > >> > > > > > to > > > >> > > > > > > > > > > integrate > > > >> > > > > > > > > > > > > with > > > >> > > > > > > > > > > > > > > > Flink > > > >> > > > > > > > > > > > > > > > > > in > > > >> > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > best > > > >> > > > > > > > > > > > > > > > > > > > > > > > > way. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Then > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > we > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > reimplement > > > the > > > >> > > > > > pulsar-flink > > > >> > > > > > > > > > > > connectors > > > >> > > > > > > > > > > > > from > > > >> > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > ground > > > >> > > > > > > > > > > > > > > > > > > > > up > > > >> > > > > > > > > > > > > > > > > > > > > > > > with > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > schema > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> bring > table > > > API > > > >> > and > > > >> > > > > > catalog > > > >> > > > > > > > API > > > >> > > > > > > > > as > > > >> > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > first-class > > > >> > > > > > > > > > > > > > > > > > > > > citizen > > > >> > > > > > > > > > > > > > > > > > > > > > > in > > > >> > > > > > > > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > integration. > > > >> With > > > >> > > that > > > >> > > > > > being > > > >> > > > > > > > > said, > > > >> > > > > > > > > > > in > > > >> > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > new > > > >> > > > > > > > > > > > > > > > > > > > > pulsar-flink > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > implementation, > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> you can > > > register > > > >> > > > pulsar > > > >> > > > > > as a > > > >> > > > > > > > > flink > > > >> > > > > > > > > > > > > catalog > > > >> > > > > > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > > > > > query / > > > >> > > > > > > > > > > > > > > > > > > > > > > process > > > >> > > > > > > > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > event > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> streams > > using > > > >> > Flink > > > >> > > > SQL. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> This is > an > > > >> example > > > >> > > > about > > > >> > > > > > how > > > >> > > > > > > > to > > > >> > > > > > > > > > use > > > >> > > > > > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > > > as > > > >> > > > > > > > > > > > > > > > a > > > >> > > > > > > > > > > > > > > > > > > Flink > > > >> > > > > > > > > > > > > > > > > > > > > > > catalog: > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Yijie has > > > also > > > >> > > > written a > > > >> > > > > > > blog > > > >> > > > > > > > > post > > > >> > > > > > > > > > > > > > > explaining > > > >> > > > > > > > > > > > > > > > > why > > > >> > > > > > > > > > > > > > > > > > we > > > >> > > > > > > > > > > > > > > > > > > > > > > > > re-implement > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > flink > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector > > > with > > > >> > Flink > > > >> > > > 1.9 > > > >> > > > > > and > > > >> > > > > > > > > what > > > >> > > > > > > > > > > are > > > >> > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > changes > > > >> > > > > > > > > > > > > > > > > > we > > > >> > > > > > > > > > > > > > > > > > > > > made > > > >> > > > > > > > > > > > > > > > > > > > > > in > > > >> > > > > > > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > new > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > connector: > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> We > believe > > > >> Pulsar > > > >> > is > > > >> > > > not > > > >> > > > > > > just > > > >> > > > > > > > a > > > >> > > > > > > > > > > simple > > > >> > > > > > > > > > > > > data > > > >> > > > > > > > > > > > > > > > sink > > > >> > > > > > > > > > > > > > > > > > or > > > >> > > > > > > > > > > > > > > > > > > > > source > > > >> > > > > > > > > > > > > > > > > > > > > > > for > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Flink. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > It > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> actually > > can > > > be > > > >> a > > > >> > > > fully > > > >> > > > > > > > > integrated > > > >> > > > > > > > > > > > > streaming > > > >> > > > > > > > > > > > > > > > > data > > > >> > > > > > > > > > > > > > > > > > > > > storage > > > >> > > > > > > > > > > > > > > > > > > > > > > for > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Flink in > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > many > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> areas > > (sink, > > > >> > source, > > > >> > > > > > > > > > schema/catalog > > > >> > > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > > state). > > > >> > > > > > > > > > > > > > > > > > The > > > >> > > > > > > > > > > > > > > > > > > > > > > > combination > > > >> > > > > > > > > > > > > > > > > > > > > > > > > of > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> and > Pulsar > > > can > > > >> > > create > > > >> > > > a > > > >> > > > > > > great > > > >> > > > > > > > > > > > streaming > > > >> > > > > > > > > > > > > > > > > warehouse > > > >> > > > > > > > > > > > > > > > > > > > > > > architecture > > > >> > > > > > > > > > > > > > > > > > > > > > > > > for > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > streaming-first, > > > >> > > > unified > > > >> > > > > > > data > > > >> > > > > > > > > > > > > processing. > > > >> > > > > > > > > > > > > > > > Since > > > >> > > > > > > > > > > > > > > > > we > > > >> > > > > > > > > > > > > > > > > > > are > > > >> > > > > > > > > > > > > > > > > > > > > > > talking > > > >> > > > > > > > > > > > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > contribute > > > >> Pulsar > > > >> > > > > > > integration > > > >> > > > > > > > to > > > >> > > > > > > > > > > Flink > > > >> > > > > > > > > > > > > here, > > > >> > > > > > > > > > > > > > > > we > > > >> > > > > > > > > > > > > > > > > > are > > > >> > > > > > > > > > > > > > > > > > > also > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > dedicated to > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> maintain, > > > >> improve > > > >> > > and > > > >> > > > > > evolve > > > >> > > > > > > > the > > > >> > > > > > > > > > > > > integration > > > >> > > > > > > > > > > > > > > > > with > > > >> > > > > > > > > > > > > > > > > > > Flink > > > >> > > > > > > > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > > > > > > > help > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > users > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> who use > > both > > > >> Flink > > > >> > > and > > > >> > > > > > > Pulsar. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hope this > > > give > > > >> > you a > > > >> > > > bit > > > >> > > > > > > more > > > >> > > > > > > > > > > > background > > > >> > > > > > > > > > > > > > > about > > > >> > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > pulsar > > > >> > > > > > > > > > > > > > > > > > > > > > > > flink > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > integration. > > > Let > > > >> > me > > > >> > > > know > > > >> > > > > > > what > > > >> > > > > > > > > are > > > >> > > > > > > > > > > your > > > >> > > > > > > > > > > > > > > > thoughts. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Thanks, > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Sijie > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Tue, > Sep > > > 3, > > > >> > 2019 > > > >> > > at > > > >> > > > > > 11:54 > > > >> > > > > > > > AM > > > >> > > > > > > > > > Yun > > > >> > > > > > > > > > > > > Tang < > > > >> > > > > > > > > > > > > > > > > > > > > > myas...@live.com> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I can > see > > > that > > > >> > > Pulsar > > > >> > > > > > > becomes > > > >> > > > > > > > > > more > > > >> > > > > > > > > > > > and > > > >> > > > > > > > > > > > > more > > > >> > > > > > > > > > > > > > > > > > popular > > > >> > > > > > > > > > > > > > > > > > > > > > > recently > > > >> > > > > > > > > > > > > > > > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > very > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> glad > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> to see > > more > > > >> > people > > > >> > > > > > willing > > > >> > > > > > > to > > > >> > > > > > > > > > > > > contribute to > > > >> > > > > > > > > > > > > > > > > Flink > > > >> > > > > > > > > > > > > > > > > > > > > > > ecosystem. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Before > any > > > >> > further > > > >> > > > > > > > discussion, > > > >> > > > > > > > > > > would > > > >> > > > > > > > > > > > > you > > > >> > > > > > > > > > > > > > > > please > > > >> > > > > > > > > > > > > > > > > > > give > > > >> > > > > > > > > > > > > > > > > > > > > some > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > explanation > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > of > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> the > > > >> relationship > > > >> > > > > between > > > >> > > > > > > this > > > >> > > > > > > > > > > thread > > > >> > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > current > > > >> > > > > > > > > > > > > > > > > > > > > existing > > > >> > > > > > > > > > > > > > > > > > > > > > > > JIRAs > > > >> > > > > > > > > > > > > > > > > > > > > > > > > of > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > pulsar > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> source > [1] > > > and > > > >> > sink > > > >> > > > [2] > > > >> > > > > > > > > > connector? > > > >> > > > > > > > > > > > > Will the > > > >> > > > > > > > > > > > > > > > > > > > > contribution > > > >> > > > > > > > > > > > > > > > > > > > > > > > > contains > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > part > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > of > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> those > PRs > > or > > > >> > > totally > > > >> > > > > > > > different > > > >> > > > > > > > > > > > > > > > implementation? > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1] > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > https://issues.apache.org/jira/browse/FLINK-9641 > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2] > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > https://issues.apache.org/jira/browse/FLINK-9168 > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > >> > > > > > > > > ________________________________ > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> From: > > Yijie > > > >> Shen > > > >> > < > > > >> > > > > > > > > > > > > > > henry.yijies...@gmail.com> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Sent: > > > Tuesday, > > > >> > > > > September > > > >> > > > > > 3, > > > >> > > > > > > > > 2019 > > > >> > > > > > > > > > > > 13:57 > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> To: > > > >> > > > > dev@flink.apache.org > > > >> > > > > > < > > > >> > > > > > > > > > > > > > > > dev@flink.apache.org > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Subject: > > > >> > [DISCUSS] > > > >> > > > > > > Contribute > > > >> > > > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > Flink > > > >> > > > > > > > > > > > > > > > > > > connector > > > >> > > > > > > > > > > > > > > > > > > > > back > > > >> > > > > > > > > > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Dear > Flink > > > >> > > Community! > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I would > > > like to > > > >> > > open > > > >> > > > > the > > > >> > > > > > > > > > discussion > > > >> > > > > > > > > > > > of > > > >> > > > > > > > > > > > > > > > > > contributing > > > >> > > > > > > > > > > > > > > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > connector > > > [0] > > > >> > back > > > >> > > to > > > >> > > > > > > Flink. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## A > brief > > > >> > > > introduction > > > >> > > > > > to > > > >> > > > > > > > > Apache > > > >> > > > > > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Apache > > > >> Pulsar[1] > > > >> > > is a > > > >> > > > > > > > > > multi-tenant, > > > >> > > > > > > > > > > > > > > > > > > high-performance > > > >> > > > > > > > > > > > > > > > > > > > > > > > > distributed > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> pub-sub > > > >> messaging > > > >> > > > > system. > > > >> > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > includes > > > >> > > > > > > > > > > > > > > > > > multiple > > > >> > > > > > > > > > > > > > > > > > > > > > features > > > >> > > > > > > > > > > > > > > > > > > > > > > > > such > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > as > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> native > > > support > > > >> > for > > > >> > > > > > multiple > > > >> > > > > > > > > > > clusters > > > >> > > > > > > > > > > > > in a > > > >> > > > > > > > > > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > > > > > > > > > > instance, > > > >> > > > > > > > > > > > > > > > > > > > > > > > with > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> seamless > > > >> > > > > geo-replication > > > >> > > > > > of > > > >> > > > > > > > > > > messages > > > >> > > > > > > > > > > > > across > > > >> > > > > > > > > > > > > > > > > > > clusters, > > > >> > > > > > > > > > > > > > > > > > > > > > very > > > >> > > > > > > > > > > > > > > > > > > > > > > > low > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > publish > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and > > > end-to-end > > > >> > > > latency, > > > >> > > > > > > > > seamless > > > >> > > > > > > > > > > > > > > scalability > > > >> > > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > > over a > > > >> > > > > > > > > > > > > > > > > > > > > > > > million > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > topics, > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and > > > guaranteed > > > >> > > > message > > > >> > > > > > > > delivery > > > >> > > > > > > > > > > with > > > >> > > > > > > > > > > > > > > > persistent > > > >> > > > > > > > > > > > > > > > > > > message > > > >> > > > > > > > > > > > > > > > > > > > > > > > storage > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> provided > > by > > > >> > Apache > > > >> > > > > > > > BookKeeper. > > > >> > > > > > > > > > > > > Nowadays, > > > >> > > > > > > > > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > > > > > > has > > > >> > > > > > > > > > > > > > > > > > > > > been > > > >> > > > > > > > > > > > > > > > > > > > > > > > > adopted > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > by > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> more and > > > more > > > >> > > > > > companies[2]. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## The > > > status > > > >> of > > > >> > > > Pulsar > > > >> > > > > > > Flink > > > >> > > > > > > > > > > > connector > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> The > Pulsar > > > >> Flink > > > >> > > > > > connector > > > >> > > > > > > we > > > >> > > > > > > > > are > > > >> > > > > > > > > > > > > planning > > > >> > > > > > > > > > > > > > > to > > > >> > > > > > > > > > > > > > > > > > > > > contribute > > > >> > > > > > > > > > > > > > > > > > > > > > is > > > >> > > > > > > > > > > > > > > > > > > > > > > > > built > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > upon > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Flink > > 1.9.0 > > > and > > > >> > > > Pulsar > > > >> > > > > > > 2.4.0. > > > >> > > > > > > > > The > > > >> > > > > > > > > > > > main > > > >> > > > > > > > > > > > > > > > features > > > >> > > > > > > > > > > > > > > > > > > are: > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar > > as > > > a > > > >> > > > streaming > > > >> > > > > > > > source > > > >> > > > > > > > > > with > > > >> > > > > > > > > > > > > > > > > exactly-once > > > >> > > > > > > > > > > > > > > > > > > > > > guarantee. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Sink > > > >> streaming > > > >> > > > > results > > > >> > > > > > to > > > >> > > > > > > > > > Pulsar > > > >> > > > > > > > > > > > with > > > >> > > > > > > > > > > > > > > > > > > at-least-once > > > >> > > > > > > > > > > > > > > > > > > > > > > > > semantics. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > (We > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> would > > update > > > >> this > > > >> > > to > > > >> > > > > > > > > exactly-once > > > >> > > > > > > > > > > as > > > >> > > > > > > > > > > > > well > > > >> > > > > > > > > > > > > > > > when > > > >> > > > > > > > > > > > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > > > > > > > > > > gets > > > >> > > > > > > > > > > > > > > > > > > > > > > > all > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > transaction > > > >> > > features > > > >> > > > > > ready > > > >> > > > > > > in > > > >> > > > > > > > > its > > > >> > > > > > > > > > > > 2.5.0 > > > >> > > > > > > > > > > > > > > > > version) > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Build > > upon > > > >> > Flink > > > >> > > > new > > > >> > > > > > > Table > > > >> > > > > > > > > API > > > >> > > > > > > > > > > Type > > > >> > > > > > > > > > > > > > > system > > > >> > > > > > > > > > > > > > > > > > > > > > (FLIP-37[3]), > > > >> > > > > > > > > > > > > > > > > > > > > > > > and > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > can > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > automatically > > > >> > > > > > (de)serialize > > > >> > > > > > > > > > > messages > > > >> > > > > > > > > > > > > with > > > >> > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > help > > > >> > > > > > > > > > > > > > > > > > > of > > > >> > > > > > > > > > > > > > > > > > > > > > > Pulsar > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > schema. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - > > Integrate > > > >> with > > > >> > > > Flink > > > >> > > > > > new > > > >> > > > > > > > > > Catalog > > > >> > > > > > > > > > > > API > > > >> > > > > > > > > > > > > > > > > > > (FLIP-30[4]), > > > >> > > > > > > > > > > > > > > > > > > > > > which > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > enables > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > the > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> use of > > > Pulsar > > > >> > > topics > > > >> > > > as > > > >> > > > > > > > tables > > > >> > > > > > > > > in > > > >> > > > > > > > > > > > > Table API > > > >> > > > > > > > > > > > > > > > as > > > >> > > > > > > > > > > > > > > > > > > well as > > > >> > > > > > > > > > > > > > > > > > > > > > SQL > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > client. > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## > > Reference > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [0] > > > >> > > > > > > > > > > > > > > > > > > > >> https://github.com/streamnative/pulsar-flink > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1] > > > >> > > > > > > > https://pulsar.apache.org/ > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2] > > > >> > > > > > > > > > > > > > > > https://pulsar.apache.org/en/powered-by/ > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [3] > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [4] > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best, > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yijie > Shen > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > >
