+1 for starting with pulsar sink and catalog. Sijie
On Tue, Sep 24, 2019 at 3:36 PM Jark Wu <imj...@gmail.com> wrote: > I agree that we can start from Pulsar sink and Pulsar Catalog. > When we finish that, maybe FLIP-27 is ready then. > > Best, > Jark > > On Tue, 24 Sep 2019 at 17:06, Becket Qin <becket....@gmail.com> wrote: > >> Hi Till, >> >> My understanding is that we are going to hold back a little bit on the >> Pulsar source connector until FLIP-27 is ready. If we check in the old >> Pulsar source connector right away, we will have to maintain that old >> connector for quite a while (a year or more), which is something we don't >> want to do. >> >> So the conclusion is following: >> - We will try our best to check in both FLIP-27 and the Pulsar source >> connector in 1.10. >> - In case there isn't enough time to get Pulsar source connector checked >> in >> 1.10, we will link prominently to the external Pulsar connector repo in >> our >> connector doc so that users can find that easily. >> >> BTW, I think we can start the work on Pulsar sink connector given that >> there is no clear plan on whether or how we are going to change the sink >> interface. >> >> @Sijie, @Yijie, what do you think? >> >> Thanks, >> >> Jiangjie (Becket) Qin >> >> On Tue, Sep 24, 2019 at 4:40 PM Till Rohrmann <trohrm...@apache.org> >> wrote: >> >> > What is the conclusion of this discussion? Are we gonna merge the old >> > source/sink or do we believe that FLIP-27 will be completed in time so >> that >> > we can merge the new source right away? >> > >> > Cheers, >> > Till >> > >> > On Tue, Sep 24, 2019 at 3:43 AM 郭士榕 <gsra...@163.com> wrote: >> > >> > > >> > > >> > > unscribe >> > > >> > > >> > > >> > > >> > > >> > > >> > > At 2019-09-24 08:53:25, "Becket Qin" <becket....@gmail.com> wrote: >> > > >Thanks, Stephan. >> > > > >> > > >Sounds good to me. We can still try our best to get new Pulsar >> connector >> > > in >> > > >Flink 1.10. In case we do not have time to do that, we will >> prominently >> > > >link the Pulsar connector from the Flink connector docs. >> > > > >> > > >Thanks, >> > > > >> > > >Jiangjie (Becket) Qin >> > > > >> > > >On Mon, Sep 23, 2019 at 4:11 PM Stephan Ewen <se...@apache.org> >> wrote: >> > > > >> > > >> Okay, I see your point, Becket. >> > > >> >> > > >> Then let us prominently link the Pulsar connector from the Flink >> > > connector >> > > >> docs then, so that users can find it easily. >> > > >> >> > > >> As soon as FLIP 27 is done, we reach out the Pulsar folks to >> > contribute >> > > a >> > > >> new connector. >> > > >> >> > > >> On Mon, Sep 23, 2019 at 3:11 AM Becket Qin <becket....@gmail.com> >> > > wrote: >> > > >> >> > > >> > Hi Stephan, >> > > >> > >> > > >> > I have no doubt about the value of adding Pulsar connector to >> Flink >> > > repo. >> > > >> > My concern is about how exactly we are going to do it. >> > > >> > >> > > >> > As mentioned before, I believe that we can handle connectors more >> > > >> > > pragmatically and less strict than the core of Flink, if it >> helps >> > > >> > unlocking >> > > >> > > users. >> > > >> > >> > > >> > I can see the benefit of being less restrict for the initial >> > connector >> > > >> code >> > > >> > adoption. However, I don't think we should be less restrict on >> the >> > > >> > maintenance commitment once the code is in Flink repo. It only >> makes >> > > >> sense >> > > >> > to check in something and ask users to use if we plan to maintain >> > it. >> > > >> > >> > > >> > If I understand correctly, the current plan so far is following: >> > > >> > 1. release 1.10 >> > > >> > - Check in Pulsar connector on old interface and label it as >> beta >> > > >> > version. >> > > >> > - encourage users to try it and report bugs. >> > > >> > 2. release 1.11 >> > > >> > - Check in Pulsar connector on new interface (a.k.a new Pulsar >> > > >> > connector) and label it as beta version >> > > >> > - Deprecate the old Pulsar connector >> > > >> > - Fix bugs reported on old Pulsar connector from release 1.10 >> > > >> > - Ask users to migrate from old Pulsar connector to new Pulsar >> > > >> connector >> > > >> > 3. release 1.12 >> > > >> > - Announce end of support for old Pulsar connector and remove >> the >> > > code >> > > >> > - Fix bugs reported on new Pulsar connector. >> > > >> > >> > > >> > If this is the plan, it seems neither Flink nor the users trying >> the >> > > old >> > > >> > Pulsar connector will benefit from this experimental old Pulsar >> > > >> connector, >> > > >> > because whatever feedbacks we got or bugs we fix on the old >> Pulsar >> > > >> > connector are immediately thrown away in one or two releases. >> > > >> > >> > > >> > If we check in the old Pulsar connector right now, the only >> option I >> > > see >> > > >> is >> > > >> > to maintain it for a while (e.g. a year or more). IMO, the >> immediate >> > > >> > deprecation and code removal hurts the users much more than >> asking >> > > them >> > > >> to >> > > >> > wait for another release. I personally think that we can avoid >> this >> > > >> > maintenance burden by going directly to the new Pulsar connector, >> > > >> > especially given that users can still use the connector even if >> they >> > > are >> > > >> > not in Flink repo. That said, I am OK with maintaining both old >> and >> > > new >> > > >> > Pulsar connector if we believe that having the Pulsar connector >> > > available >> > > >> > right now in Flink repo is more important. >> > > >> > >> > > >> > Thanks, >> > > >> > >> > > >> > Jiangjie (Becket) Qin >> > > >> > >> > > >> > On Sun, Sep 22, 2019 at 9:10 PM Stephan Ewen <se...@apache.org> >> > > wrote: >> > > >> > >> > > >> > > My assumption is as Sijie's, that once the connector is either >> > part >> > > of >> > > >> > > Flink, or part of the streamnative repo. No double maintenance. >> > > >> > > >> > > >> > > I feel this discussion is very much caught in problems that are >> > all >> > > >> > > solvable if we want to solve them. >> > > >> > > Maybe we can think what our goal for users and the communities >> is? >> > > >> > > >> > > >> > > - Do we want to help build a relationship between the Pulsar >> and >> > > >> Flink >> > > >> > > open source communities? >> > > >> > > - Will users find a connector in the streamnative repository? >> > > >> > > - Will users trust a connector that is not part of Flink as >> > much? >> > > >> > > >> > > >> > > And then decide what is best according to the overall goals >> there. >> > > >> > > As mentioned before, I believe that we can handle connectors >> more >> > > >> > > pragmatically and less strict than the core of Flink, if it >> helps >> > > >> > unlocking >> > > >> > > users. >> > > >> > > >> > > >> > > Best, >> > > >> > > Stephan >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > On Fri, Sep 20, 2019 at 2:10 PM Sijie Guo <guosi...@gmail.com> >> > > wrote: >> > > >> > > >> > > >> > > > Thanks Becket. >> > > >> > > > >> > > >> > > > I think it is better for the Flink community to judge the >> > > benefits of >> > > >> > > doing >> > > >> > > > this. I was trying to provide some views from outsiders. >> > > >> > > > >> > > >> > > > Thanks, >> > > >> > > > Sijie >> > > >> > > > >> > > >> > > > On Fri, Sep 20, 2019 at 10:25 AM Becket Qin < >> > becket....@gmail.com >> > > > >> > > >> > > wrote: >> > > >> > > > >> > > >> > > > > Hi Sijie, >> > > >> > > > > >> > > >> > > > > Yes, we will have to support existing old connectors and >> new >> > > >> > connectors >> > > >> > > > in >> > > >> > > > > parallel for a while. We have to take that maintenance >> > overhead >> > > >> > because >> > > >> > > > > existing connectors have been used by the users for a long >> > > time. I >> > > >> > > guess >> > > >> > > > It >> > > >> > > > > may take at least a year for us to fully remove the old >> > > connectors. >> > > >> > > > > >> > > >> > > > > Process wise, we can do the same for Pulsar connector. But >> I >> > am >> > > not >> > > >> > > sure >> > > >> > > > if >> > > >> > > > > we want to have the same burden on Pulsar connector, and I >> > would >> > > >> like >> > > >> > > to >> > > >> > > > > understand the benefit of doing that. >> > > >> > > > > >> > > >> > > > > For users, the benefit of having the old Pulsar connector >> > > checked >> > > >> in >> > > >> > > > seems >> > > >> > > > > limited because 1) that code base will be immediately >> > > deprecated in >> > > >> > the >> > > >> > > > > next release in 3-4 months; 2) users can always use it >> even if >> > > it >> > > >> is >> > > >> > > not >> > > >> > > > in >> > > >> > > > > the Flink code base. Admittedly it is not as convenient as >> > > having >> > > >> it >> > > >> > in >> > > >> > > > > Flink code base, but doesn't seem super either. And after >> 3-4 >> > > >> months, >> > > >> > > > users >> > > >> > > > > can just use the new connector in Flink repo. >> > > >> > > > > >> > > >> > > > > For Flink developers, the old connector code base is not >> > > something >> > > >> > that >> > > >> > > > we >> > > >> > > > > want to evolve later. Instead, these code will be >> deprecated >> > and >> > > >> > > > > removed. So why do we want to get a beta version out to >> > attract >> > > >> > people >> > > >> > > to >> > > >> > > > > use something we don't want to maintain? >> > > >> > > > > >> > > >> > > > > Thanks, >> > > >> > > > > >> > > >> > > > > Jiangjie (Becket) Qin >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > On Fri, Sep 20, 2019 at 10:12 AM Sijie Guo < >> > guosi...@gmail.com> >> > > >> > wrote: >> > > >> > > > > >> > > >> > > > > > Thanks everyone here. Sorry for jumping into the >> discussion >> > > here. >> > > >> > > > > > >> > > >> > > > > > I am not very familiar about the deprecation process in >> > Flink. >> > > >> If I >> > > >> > > > > > misunderstood the process, please fix me. >> > > >> > > > > > >> > > >> > > > > > As far as I understand, FLIP-27 is introducing a new >> unified >> > > API >> > > >> > for >> > > >> > > > > > connectors. After it introduces the new API >> > > >> > > > > > and before moving all the existing connectors from old >> API >> > to >> > > new >> > > >> > > API, >> > > >> > > > > both >> > > >> > > > > > old ApI and new API will co-exist >> > > >> > > > > > for a while until Flink moves all existing connectors to >> new >> > > API. >> > > >> > So >> > > >> > > > the >> > > >> > > > > > Pulsar connector (using old API) can >> > > >> > > > > > follow the deprecation process with other connector using >> > old >> > > API >> > > >> > and >> > > >> > > > the >> > > >> > > > > > deprecation of old API, no? >> > > >> > > > > > >> > > >> > > > > > If that's the case, I think contributing the current >> > connector >> > > >> back >> > > >> > > to >> > > >> > > > > > Flink rather than maintaining it outside Flink >> > > >> > > > > > would provide a bit more benefits. We can deprecate the >> > > existing >> > > >> > > > > > streamnative/pulsar-flink repo and point the users >> > > >> > > > > > to use the connector in Flink repo. So all the review >> > > processes >> > > >> > will >> > > >> > > > > happen >> > > >> > > > > > within Flink for both old connector and >> > > >> > > > > > new connector. It also reduces the confusions for the >> users >> > as >> > > >> the >> > > >> > > > > > documentation and code base happen in one place. >> > > >> > > > > > >> > > >> > > > > > Thoughts? >> > > >> > > > > > >> > > >> > > > > > - Sijie >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > On Fri, Sep 20, 2019 at 12:53 AM Becket Qin < >> > > >> becket....@gmail.com> >> > > >> > > > > wrote: >> > > >> > > > > > >> > > >> > > > > > > Thanks for the explanation, Stephan. I have a few >> > questions >> > > / >> > > >> > > > thoughts. >> > > >> > > > > > > >> > > >> > > > > > > So that means we will remove the old connector without >> a >> > > major >> > > >> > > > version >> > > >> > > > > > > bump, is that correct? >> > > >> > > > > > > >> > > >> > > > > > > I am not 100% sure if mixing 1.10 connectors with 1.11 >> > > >> connectors >> > > >> > > > will >> > > >> > > > > > > always work because we saw some dependency class >> > collisions >> > > in >> > > >> > the >> > > >> > > > > past. >> > > >> > > > > > To >> > > >> > > > > > > make it safe we may have to maintain the old code for >> one >> > > more >> > > >> > > > release. >> > > >> > > > > > > >> > > >> > > > > > > To be honest I am still wondering if we have to put the >> > old >> > > >> > > connector >> > > >> > > > > in >> > > >> > > > > > > Flink repo. if we check in the old connector to Flink. >> We >> > > will >> > > >> > end >> > > >> > > up >> > > >> > > > > in >> > > >> > > > > > > the following situation: >> > > >> > > > > > > 1. Old connector in streamnative/pulsar-flink repo. >> > > >> > > > > > > 2. Old connector in Flink Repo, which may be different >> > from >> > > the >> > > >> > one >> > > >> > > > in >> > > >> > > > > > > Pulsar repo. (Added in 1.10, deprecated in 1.11, >> removed >> > in >> > > >> 1.12) >> > > >> > > > > > > 3. New connector in Flink Repo. >> > > >> > > > > > > >> > > >> > > > > > > We need to think about how to make the users in each >> case >> > > >> happy. >> > > >> > > > > > > - For users of (1), I assume Sijie and Yijie will have >> to >> > > >> > maintain >> > > >> > > > the >> > > >> > > > > > code >> > > >> > > > > > > a bit longer for its own compatibility even after we >> have >> > > (2). >> > > >> In >> > > >> > > > that >> > > >> > > > > > > case, bugs found in old connector may or may not need >> to >> > be >> > > >> fixed >> > > >> > > in >> > > >> > > > > both >> > > >> > > > > > > Flink and the streamnative/pulsar-flink repo. >> > > >> > > > > > > - For users of (2), will we provide bug fixes? If we >> do, >> > it >> > > >> will >> > > >> > > be a >> > > >> > > > > > > little awkward because those bug fixes will be >> immediately >> > > >> > > deprecated >> > > >> > > > > in >> > > >> > > > > > > 1.11, and removed in 1.12. So we are essentially asking >> > > users >> > > >> to >> > > >> > > > > migrate >> > > >> > > > > > > away from the bug fix. After Flink 1.12, users may >> still >> > > have >> > > >> to >> > > >> > > > switch >> > > >> > > > > > to >> > > >> > > > > > > use (3) due to the potential dependency class conflicts >> > > >> mentioned >> > > >> > > > > above. >> > > >> > > > > > > - Users of (3) have a much easier life and don't need >> to >> > > worry >> > > >> > too >> > > >> > > > > much. >> > > >> > > > > > > >> > > >> > > > > > > The above story seems a little complicated to tell. I >> > think >> > > it >> > > >> > will >> > > >> > > > be >> > > >> > > > > > much >> > > >> > > > > > > easier to not have (2) at all. >> > > >> > > > > > > 1. Old connector in streamnative/pulsar-flink repo. >> > > >> > > > > > > 3. New connector in Flink Repo. >> > > >> > > > > > > >> > > >> > > > > > > - Old connector will only be maintained in >> > > >> > > streamnative/pulsar-flink >> > > >> > > > > repo >> > > >> > > > > > > until it is fully deprecated. Users can always use the >> > > existing >> > > >> > > > Pulsar >> > > >> > > > > > > connector in that repo. >> > > >> > > > > > > - New connector will be in Flink repo and maintained >> like >> > > the >> > > >> > other >> > > >> > > > > > > connectors. >> > > >> > > > > > > >> > > >> > > > > > > This seems much simpler for users understand and they >> will >> > > not >> > > >> be >> > > >> > > > > blocked >> > > >> > > > > > > from using the old connector. If the concern is about >> the >> > > >> quality >> > > >> > > of >> > > >> > > > > the >> > > >> > > > > > > connector in streamnative/pulsar-flink repo, is it >> enough >> > > for >> > > >> us >> > > >> > > just >> > > >> > > > > to >> > > >> > > > > > > review the code in streamnative/pulsar-flink connector >> to >> > > make >> > > >> > sure >> > > >> > > > it >> > > >> > > > > > > looks good from Flink's perspective? >> > > >> > > > > > > >> > > >> > > > > > > What do you think? >> > > >> > > > > > > >> > > >> > > > > > > Thanks, >> > > >> > > > > > > >> > > >> > > > > > > Jiangjie (Becket) Qin >> > > >> > > > > > > >> > > >> > > > > > > >> > > >> > > > > > > On Thu, Sep 19, 2019 at 6:58 PM Stephan Ewen < >> > > se...@apache.org >> > > >> > >> > > >> > > > wrote: >> > > >> > > > > > > >> > > >> > > > > > > > My take would be the following: >> > > >> > > > > > > > >> > > >> > > > > > > > - If we merge the connector now and replace it >> with a >> > > >> FLIP-27 >> > > >> > > > > version >> > > >> > > > > > > > before the 1.10 release, then we need no deprecation >> > > process >> > > >> > > > > > > > - If we don't manage to replace it with a FLIP-27 >> > > version >> > > >> > > before >> > > >> > > > > the >> > > >> > > > > > > 1.10 >> > > >> > > > > > > > release, than it is good that we have the other >> version, >> > > so >> > > >> no >> > > >> > > > users >> > > >> > > > > > get >> > > >> > > > > > > > blocked. >> > > >> > > > > > > > >> > > >> > > > > > > > In the latter case we can see how we want to do it. >> > > Immediate >> > > >> > > > removal >> > > >> > > > > > of >> > > >> > > > > > > > the old version or deprecation label and keeping it >> for >> > > one >> > > >> > more >> > > >> > > > > > release. >> > > >> > > > > > > > Given that you should be able to use a Flink 1.10 >> > > connector >> > > >> > with >> > > >> > > > > Flink >> > > >> > > > > > > 1.11 >> > > >> > > > > > > > as well (stable public APIs) there is also a >> workaround >> > if >> > > >> you >> > > >> > > need >> > > >> > > > > an >> > > >> > > > > > > old >> > > >> > > > > > > > connector in a newer version. So immediate removal >> might >> > > even >> > > >> > be >> > > >> > > > > > > feasible. >> > > >> > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > > > On Thu, Sep 19, 2019 at 11:09 AM Becket Qin < >> > > >> > > becket....@gmail.com> >> > > >> > > > > > > wrote: >> > > >> > > > > > > > >> > > >> > > > > > > > > Hi Stephan, >> > > >> > > > > > > > > >> > > >> > > > > > > > > Thanks for the clarification. I completely agree >> with >> > > you >> > > >> and >> > > >> > > > > Thomas >> > > >> > > > > > on >> > > >> > > > > > > > the >> > > >> > > > > > > > > process of adding connectors to Flink repo. >> However, I >> > > am >> > > >> > > > wondering >> > > >> > > > > > > what >> > > >> > > > > > > > is >> > > >> > > > > > > > > the deprecation process? Given the main concern >> here >> > was >> > > >> that >> > > >> > > we >> > > >> > > > > may >> > > >> > > > > > > have >> > > >> > > > > > > > > to maintain two Pulsar connector code bases until >> the >> > > old >> > > >> one >> > > >> > > is >> > > >> > > > > > > removed >> > > >> > > > > > > > > from the repo, it would be good to know how long we >> > > have to >> > > >> > do >> > > >> > > > > that. >> > > >> > > > > > > > > >> > > >> > > > > > > > > Thanks, >> > > >> > > > > > > > > >> > > >> > > > > > > > > Jiangjie (Becket) Qin >> > > >> > > > > > > > > >> > > >> > > > > > > > > On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen < >> > > >> > se...@apache.org >> > > >> > > > >> > > >> > > > > > wrote: >> > > >> > > > > > > > > >> > > >> > > > > > > > > > Some quick thoughts on the connector contribution >> > > >> process. >> > > >> > I >> > > >> > > > > > > basically >> > > >> > > > > > > > > > reiterate here what Thomas mentioned in another >> > thread >> > > >> > about >> > > >> > > > the >> > > >> > > > > > > > Kinesis >> > > >> > > > > > > > > > connector. >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > For connectors, we should favor a low-overhead >> > > >> contribution >> > > >> > > > > > process, >> > > >> > > > > > > > and >> > > >> > > > > > > > > > accept user code and changes more readily than in >> > the >> > > >> core >> > > >> > > > > system. >> > > >> > > > > > > > > > That is because connectors have both a big >> variety >> > of >> > > >> > > scenarios >> > > >> > > > > > they >> > > >> > > > > > > > get >> > > >> > > > > > > > > > used in (only through use and many small >> > > contributions do >> > > >> > > they >> > > >> > > > > > become >> > > >> > > > > > > > > > really useful over time) and at the same time, >> and >> > > >> > committers >> > > >> > > > do >> > > >> > > > > > not >> > > >> > > > > > > > use >> > > >> > > > > > > > > > the connector themselves and usually cannot >> foresee >> > > too >> > > >> > well >> > > >> > > > what >> > > >> > > > > > is >> > > >> > > > > > > > > > needed. >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > Further more, a missing connector (or connector >> > > feature) >> > > >> is >> > > >> > > > > often a >> > > >> > > > > > > > > bigger >> > > >> > > > > > > > > > show stopper for users than a missing API or >> system >> > > >> > feature. >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > Along these lines of thougt, the conclusion >> would be >> > > to >> > > >> > take >> > > >> > > > the >> > > >> > > > > > > Pulsar >> > > >> > > > > > > > > > connector now, focus the review on >> > > >> legal/dependencies/rough >> > > >> > > > code >> > > >> > > > > > > style >> > > >> > > > > > > > > and >> > > >> > > > > > > > > > conventions, label it as "beta" (in the sense of >> > "new >> > > >> code" >> > > >> > > > that >> > > >> > > > > is >> > > >> > > > > > > > "not >> > > >> > > > > > > > > > yet tested through longer use") and go ahead. And >> > then >> > > >> > evolve >> > > >> > > > it >> > > >> > > > > > > > quickly >> > > >> > > > > > > > > > without putting formal blockers in the way, >> meaning >> > > also >> > > >> > > > adding a >> > > >> > > > > > new >> > > >> > > > > > > > > FLIP >> > > >> > > > > > > > > > 27 version when it is there. >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > Best, >> > > >> > > > > > > > > > Stephan >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin < >> > > >> > > > becket....@gmail.com >> > > >> > > > > > >> > > >> > > > > > > > wrote: >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > > Hi Yijie, >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > Could you please follow the FLIP process to >> start >> > a >> > > new >> > > >> > > FLIP >> > > >> > > > > > > > > [DISCUSSION] >> > > >> > > > > > > > > > > thread in the mailing list? >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > > >> > > >> > > > > > >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > >> > >> > > >> >> > > >> > >> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > I see two FLIP-69 discussion in the mailing >> list >> > > now. >> > > >> So >> > > >> > > > there >> > > >> > > > > > is a >> > > >> > > > > > > > > FLIP >> > > >> > > > > > > > > > > number collision. Can you change the FLIP >> number >> > to >> > > 72? >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > Thanks, >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > Jiangjie (Becket) Qin >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong < >> > > >> > > > > walter...@gmail.com> >> > > >> > > > > > > > > wrote: >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > > Hi Yijie, >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > Thanks for sharing the pulsar FLIP. >> > > >> > > > > > > > > > > > Would you mind enabling comments/suggestions >> on >> > > the >> > > >> > > google >> > > >> > > > > doc >> > > >> > > > > > > > link? >> > > >> > > > > > > > > > This >> > > >> > > > > > > > > > > > way the contributors from the community can >> > > comment >> > > >> on >> > > >> > > the >> > > >> > > > > doc. >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > Best, >> > > >> > > > > > > > > > > > Rong >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen < >> > > >> > > > > > > > > henry.yijies...@gmail.com> >> > > >> > > > > > > > > > > > wrote: >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > > Hello everyone, >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > I've drafted a FLIP that describes the >> current >> > > >> design >> > > >> > > of >> > > >> > > > > the >> > > >> > > > > > > > Pulsar >> > > >> > > > > > > > > > > > > connector: >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > > >> > > >> > > > > > >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > >> > >> > > >> >> > > >> > >> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit# >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > Please take a look and let me know what you >> > > think. >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > Thanks, >> > > >> > > > > > > > > > > > > Yijie >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong >> < >> > > >> > > > > > > walter...@gmail.com> >> > > >> > > > > > > > > > > wrote: >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > Hi All, >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > Sorry for joining the discussion late and >> > > thanks >> > > >> > > Yijie >> > > >> > > > & >> > > >> > > > > > > Sijie >> > > >> > > > > > > > > for >> > > >> > > > > > > > > > > > > driving >> > > >> > > > > > > > > > > > > > the discussion. >> > > >> > > > > > > > > > > > > > I also think the Pulsar connector would >> be a >> > > very >> > > >> > > > > valuable >> > > >> > > > > > > > > addition >> > > >> > > > > > > > > > > to >> > > >> > > > > > > > > > > > > > Flink. I can also help out a bit on the >> > review >> > > >> side >> > > >> > > :-) >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > Regarding the timeline, I also share >> > concerns >> > > >> with >> > > >> > > > Becket >> > > >> > > > > > on >> > > >> > > > > > > > the >> > > >> > > > > > > > > > > > > > relationship between the new Pulsar >> > connector >> > > and >> > > >> > > > > FLIP-27. >> > > >> > > > > > > > > > > > > > There's also another discussion just >> started >> > > by >> > > >> > > Stephan >> > > >> > > > > on >> > > >> > > > > > > > > dropping >> > > >> > > > > > > > > > > > Kafka >> > > >> > > > > > > > > > > > > > 9/10 support on next Flink release [1]. >> > > Although >> > > >> > the >> > > >> > > > > > > situation >> > > >> > > > > > > > > is >> > > >> > > > > > > > > > > > > somewhat >> > > >> > > > > > > > > > > > > > different, and Kafka 9/10 connector has >> been >> > > in >> > > >> > Flink >> > > >> > > > for >> > > >> > > > > > > > almost >> > > >> > > > > > > > > > 3-4 >> > > >> > > > > > > > > > > > > years, >> > > >> > > > > > > > > > > > > > based on the discussion I am not sure if >> a >> > > major >> > > >> > > > version >> > > >> > > > > > > > release >> > > >> > > > > > > > > > is a >> > > >> > > > > > > > > > > > > > requirement for removing old connector >> > > supports. >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > I think there shouldn't be a blocker if >> we >> > > agree >> > > >> > the >> > > >> > > > old >> > > >> > > > > > > > > connector >> > > >> > > > > > > > > > > will >> > > >> > > > > > > > > > > > > be >> > > >> > > > > > > > > > > > > > removed once FLIP-27 based Pulsar >> connector >> > is >> > > >> > there. >> > > >> > > > As >> > > >> > > > > > > > Stephan >> > > >> > > > > > > > > > > > stated, >> > > >> > > > > > > > > > > > > it >> > > >> > > > > > > > > > > > > > is easier to contribute the source sooner >> > and >> > > >> > adjust >> > > >> > > it >> > > >> > > > > > > later. >> > > >> > > > > > > > > > > > > > We should also ensure we clearly >> communicate >> > > the >> > > >> > > > message: >> > > >> > > > > > for >> > > >> > > > > > > > > > > example, >> > > >> > > > > > > > > > > > > > putting an experimental flag on the >> > pre-FLIP27 >> > > >> > > > connector >> > > >> > > > > > page >> > > >> > > > > > > > of >> > > >> > > > > > > > > > the >> > > >> > > > > > > > > > > > > > website, documentations, etc. Any other >> > > thoughts? >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > -- >> > > >> > > > > > > > > > > > > > Rong >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > [1] >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > > >> > > >> > > > > > >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > >> > >> > > >> >> > > >> > >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket >> Qin < >> > > >> > > > > > > > becket....@gmail.com >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > > > wrote: >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > Technically speaking, removing the old >> > > >> connector >> > > >> > > code >> > > >> > > > > is >> > > >> > > > > > a >> > > >> > > > > > > > > > > backwards >> > > >> > > > > > > > > > > > > > > incompatible change which requires a >> major >> > > >> > version >> > > >> > > > > bump, >> > > >> > > > > > > i.e. >> > > >> > > > > > > > > > Flink >> > > >> > > > > > > > > > > > > 2.x. >> > > >> > > > > > > > > > > > > > > Given that we don't have a clear plan >> on >> > > when >> > > >> to >> > > >> > > have >> > > >> > > > > the >> > > >> > > > > > > > next >> > > >> > > > > > > > > > > major >> > > >> > > > > > > > > > > > > > > version release, it seems unclear how >> long >> > > the >> > > >> > old >> > > >> > > > > > > connector >> > > >> > > > > > > > > code >> > > >> > > > > > > > > > > > will >> > > >> > > > > > > > > > > > > be >> > > >> > > > > > > > > > > > > > > there if we check it in right now. Or >> will >> > > we >> > > >> > > remove >> > > >> > > > > the >> > > >> > > > > > > old >> > > >> > > > > > > > > > > > connector >> > > >> > > > > > > > > > > > > > > without a major version bump? In any >> case, >> > > it >> > > >> > > sounds >> > > >> > > > > not >> > > >> > > > > > > > quite >> > > >> > > > > > > > > > user >> > > >> > > > > > > > > > > > > > > friendly to the those who might use the >> > old >> > > >> > Pulsar >> > > >> > > > > > > > connector. I >> > > >> > > > > > > > > > am >> > > >> > > > > > > > > > > > not >> > > >> > > > > > > > > > > > > sure >> > > >> > > > > > > > > > > > > > > if it is worth these potential >> problems in >> > > >> order >> > > >> > to >> > > >> > > > > have >> > > >> > > > > > > the >> > > >> > > > > > > > > > Pulsar >> > > >> > > > > > > > > > > > > source >> > > >> > > > > > > > > > > > > > > connector checked in one or two months >> > > earlier. >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > Thanks, >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > Jiangjie (Becket) Qin >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan >> > > Ewen < >> > > >> > > > > > > > se...@apache.org >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > > > wrote: >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > Agreed, if we check in the old code, >> we >> > > >> should >> > > >> > > make >> > > >> > > > > it >> > > >> > > > > > > > clear >> > > >> > > > > > > > > > that >> > > >> > > > > > > > > > > > it >> > > >> > > > > > > > > > > > > will >> > > >> > > > > > > > > > > > > > > > be removed as soon as the FLIP-27 >> based >> > > >> version >> > > >> > > of >> > > >> > > > > the >> > > >> > > > > > > > > > connector >> > > >> > > > > > > > > > > is >> > > >> > > > > > > > > > > > > > > there. >> > > >> > > > > > > > > > > > > > > > We should not commit to maintaining >> the >> > > old >> > > >> > > > version, >> > > >> > > > > > that >> > > >> > > > > > > > > would >> > > >> > > > > > > > > > > be >> > > >> > > > > > > > > > > > > indeed >> > > >> > > > > > > > > > > > > > > > too much overhead. >> > > >> > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM >> Becket >> > > Qin < >> > > >> > > > > > > > > > becket....@gmail.com >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > > wrote: >> > > >> > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > Hi Stephan, >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > Thanks for the volunteering to >> help. >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > Yes, the overhead would just be >> review >> > > >> > > capacity. >> > > >> > > > In >> > > >> > > > > > > > fact, I >> > > >> > > > > > > > > > am >> > > >> > > > > > > > > > > > not >> > > >> > > > > > > > > > > > > > > > worrying >> > > >> > > > > > > > > > > > > > > > > too much about the review capacity. >> > > That is >> > > >> > > just >> > > >> > > > a >> > > >> > > > > > one >> > > >> > > > > > > > time >> > > >> > > > > > > > > > > cost. >> > > >> > > > > > > > > > > > > My >> > > >> > > > > > > > > > > > > > > > > concern is mainly about the long >> term >> > > >> burden. >> > > >> > > > > Assume >> > > >> > > > > > we >> > > >> > > > > > > > > have >> > > >> > > > > > > > > > > new >> > > >> > > > > > > > > > > > > source >> > > >> > > > > > > > > > > > > > > > > interface ready in 1.10 with newly >> > added >> > > >> > Pulsar >> > > >> > > > > > > > connectors >> > > >> > > > > > > > > in >> > > >> > > > > > > > > > > old >> > > >> > > > > > > > > > > > > > > > > interface. Later on if we migrate >> > > Pulsar to >> > > >> > new >> > > >> > > > > > source >> > > >> > > > > > > > > > > interface, >> > > >> > > > > > > > > > > > > the >> > > >> > > > > > > > > > > > > > > old >> > > >> > > > > > > > > > > > > > > > > Pulsar connector might be >> deprecated >> > > almost >> > > >> > > > > > immediately >> > > >> > > > > > > > > after >> > > >> > > > > > > > > > > > > checked >> > > >> > > > > > > > > > > > > > > in, >> > > >> > > > > > > > > > > > > > > > > but we may still have to maintain >> two >> > > code >> > > >> > > bases. >> > > >> > > > > For >> > > >> > > > > > > the >> > > >> > > > > > > > > > > > existing >> > > >> > > > > > > > > > > > > > > > > connectors, we have to do that >> > anyways. >> > > But >> > > >> > it >> > > >> > > > > would >> > > >> > > > > > be >> > > >> > > > > > > > > good >> > > >> > > > > > > > > > to >> > > >> > > > > > > > > > > > > avoid >> > > >> > > > > > > > > > > > > > > > > introducing a new connector with >> the >> > > same >> > > >> > > > problem. >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > Thanks, >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM >> > Stephan >> > > >> Ewen >> > > >> > < >> > > >> > > > > > > > > > se...@apache.org >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > > wrote: >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > Hi all! >> > > >> > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > Nice to see this lively >> discussion >> > > about >> > > >> > the >> > > >> > > > > Pulsar >> > > >> > > > > > > > > > > connector. >> > > >> > > > > > > > > > > > > > > > > > Some thoughts on the open >> questions: >> > > >> > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > ## Contribute to Flink or >> maintain >> > as >> > > a >> > > >> > > > community >> > > >> > > > > > > > package >> > > >> > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > Looks like the discussion is more >> > > going >> > > >> > > towards >> > > >> > > > > > > > > > > contribution. I >> > > >> > > > > > > > > > > > > think >> > > >> > > > > > > > > > > > > > > > > that >> > > >> > > > > > > > > > > > > > > > > > is good, especially if we think >> that >> > > we >> > > >> > want >> > > >> > > to >> > > >> > > > > > > build a >> > > >> > > > > > > > > > > > similarly >> > > >> > > > > > > > > > > > > > > deep >> > > >> > > > > > > > > > > > > > > > > > integration with Pulsar as we >> have >> > for >> > > >> > > example >> > > >> > > > > with >> > > >> > > > > > > > > Kafka. >> > > >> > > > > > > > > > > The >> > > >> > > > > > > > > > > > > > > > connector >> > > >> > > > > > > > > > > > > > > > > > already looks like a more >> thorough >> > > >> > connector >> > > >> > > > than >> > > >> > > > > > > many >> > > >> > > > > > > > > > others >> > > >> > > > > > > > > > > > we >> > > >> > > > > > > > > > > > > have >> > > >> > > > > > > > > > > > > > > > in >> > > >> > > > > > > > > > > > > > > > > > the repository. >> > > >> > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > With either a repo split, or the >> new >> > > >> build >> > > >> > > > > system, >> > > >> > > > > > I >> > > >> > > > > > > > hope >> > > >> > > > > > > > > > > that >> > > >> > > > > > > > > > > > > the >> > > >> > > > > > > > > > > > > > > > build >> > > >> > > > > > > > > > > > > > > > > > overhead is not a problem. >> > > >> > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > ## Committer Support >> > > >> > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > Becket offered some help >> already, I >> > > can >> > > >> > also >> > > >> > > > > help a >> > > >> > > > > > > > bit. >> > > >> > > > > > > > > I >> > > >> > > > > > > > > > > hope >> > > >> > > > > > > > > > > > > that >> > > >> > > > > > > > > > > > > > > > > > between us, we can cover this. >> > > >> > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > ## Contribute now, or wait for >> > FLIP-27 >> > > >> > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > As Becket said, FLIP-27 is >> actually >> > > >> making >> > > >> > > some >> > > >> > > > > > > PoC-ing >> > > >> > > > > > > > > > > > > progress, but >> > > >> > > > > > > > > > > > > > > > > will >> > > >> > > > > > > > > > > > > > > > > > take 2 more months, I would >> > estimate, >> > > >> > before >> > > >> > > it >> > > >> > > > > is >> > > >> > > > > > > > fully >> > > >> > > > > > > > > > > > > available. >> > > >> > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > If we want to be on the safe side >> > with >> > > >> the >> > > >> > > > > > > > contribution, >> > > >> > > > > > > > > we >> > > >> > > > > > > > > > > > > should >> > > >> > > > > > > > > > > > > > > > > > contribute the source sooner and >> > > adjust >> > > >> it >> > > >> > > > later. >> > > >> > > > > > > That >> > > >> > > > > > > > > > would >> > > >> > > > > > > > > > > > also >> > > >> > > > > > > > > > > > > > > help >> > > >> > > > > > > > > > > > > > > > us >> > > >> > > > > > > > > > > > > > > > > > in case things get crazy towards >> the >> > > 1.10 >> > > >> > > > feature >> > > >> > > > > > > > freeze >> > > >> > > > > > > > > > and >> > > >> > > > > > > > > > > it >> > > >> > > > > > > > > > > > > would >> > > >> > > > > > > > > > > > > > > > be >> > > >> > > > > > > > > > > > > > > > > > hard to find time to review the >> new >> > > >> > changes. >> > > >> > > > > > > > > > > > > > > > > > What would be the overhead of >> > > >> contributing >> > > >> > > now? >> > > >> > > > > > Given >> > > >> > > > > > > > > that >> > > >> > > > > > > > > > > the >> > > >> > > > > > > > > > > > > code >> > > >> > > > > > > > > > > > > > > is >> > > >> > > > > > > > > > > > > > > > > > already there, it looks like it >> > would >> > > be >> > > >> > only >> > > >> > > > > > review >> > > >> > > > > > > > > > > capacity, >> > > >> > > > > > > > > > > > > right? >> > > >> > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > Best, >> > > >> > > > > > > > > > > > > > > > > > Stephan >> > > >> > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM >> > Yijie >> > > >> > Shen < >> > > >> > > > > > > > > > > > > > > henry.yijies...@gmail.com >> > > >> > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > wrote: >> > > >> > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > Hi everyone! >> > > >> > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > Thanks for your attention and >> the >> > > >> > promotion >> > > >> > > > of >> > > >> > > > > > this >> > > >> > > > > > > > > work. >> > > >> > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > We will prepare a FLIP as soon >> as >> > > >> > possible >> > > >> > > > for >> > > >> > > > > > more >> > > >> > > > > > > > > > > specific >> > > >> > > > > > > > > > > > > > > > > discussions. >> > > >> > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > For FLIP-27, it seems that we >> have >> > > not >> > > >> > > > reached >> > > >> > > > > a >> > > >> > > > > > > > > > consensus. >> > > >> > > > > > > > > > > > > > > > Therefore, >> > > >> > > > > > > > > > > > > > > > > > > I will explain all the >> > > functionalities >> > > >> of >> > > >> > > the >> > > >> > > > > > > > existing >> > > >> > > > > > > > > > > > > connector in >> > > >> > > > > > > > > > > > > > > > > > > the FLIP (including Source, >> Sink, >> > > and >> > > >> > > > Catalog) >> > > >> > > > > to >> > > >> > > > > > > > > > continue >> > > >> > > > > > > > > > > > our >> > > >> > > > > > > > > > > > > > > > > > > discussions in FLIP. >> > > >> > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > Thanks for your kind help. >> > > >> > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > Best, >> > > >> > > > > > > > > > > > > > > > > > > Yijie >> > > >> > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM >> > > Becket >> > > >> > Qin >> > > >> > > < >> > > >> > > > > > > > > > > > > becket....@gmail.com> >> > > >> > > > > > > > > > > > > > > > > wrote: >> > > >> > > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > > Hi Sijie, >> > > >> > > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > > If we agree that the goal is >> to >> > > have >> > > >> > > Pulsar >> > > >> > > > > > > > connector >> > > >> > > > > > > > > > in >> > > >> > > > > > > > > > > > > 1.10, >> > > >> > > > > > > > > > > > > > > how >> > > >> > > > > > > > > > > > > > > > > > about >> > > >> > > > > > > > > > > > > > > > > > > we >> > > >> > > > > > > > > > > > > > > > > > > > do the following: >> > > >> > > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar >> > > >> connector >> > > >> > > to >> > > >> > > > > > Flink >> > > >> > > > > > > > main >> > > >> > > > > > > > > > > repo >> > > >> > > > > > > > > > > > > as it >> > > >> > > > > > > > > > > > > > > > is a >> > > >> > > > > > > > > > > > > > > > > > new >> > > >> > > > > > > > > > > > > > > > > > > > public interface to Flink >> main >> > > repo. >> > > >> > > > > > > > > > > > > > > > > > > > 1. Start to review the Pulsar >> > sink >> > > >> > right >> > > >> > > > away >> > > >> > > > > > as >> > > >> > > > > > > > > there >> > > >> > > > > > > > > > is >> > > >> > > > > > > > > > > > no >> > > >> > > > > > > > > > > > > > > change >> > > >> > > > > > > > > > > > > > > > > to >> > > >> > > > > > > > > > > > > > > > > > > the >> > > >> > > > > > > > > > > > > > > > > > > > sink interface so far. >> > > >> > > > > > > > > > > > > > > > > > > > 2. Wait a little bit on >> FLIP-27. >> > > >> Flink >> > > >> > > 1.10 >> > > >> > > > > is >> > > >> > > > > > > > going >> > > >> > > > > > > > > to >> > > >> > > > > > > > > > > be >> > > >> > > > > > > > > > > > > code >> > > >> > > > > > > > > > > > > > > > > freeze >> > > >> > > > > > > > > > > > > > > > > > in >> > > >> > > > > > > > > > > > > > > > > > > > late Nov and let's say we >> give a >> > > >> month >> > > >> > to >> > > >> > > > the >> > > >> > > > > > > > > > development >> > > >> > > > > > > > > > > > and >> > > >> > > > > > > > > > > > > > > > review >> > > >> > > > > > > > > > > > > > > > > of >> > > >> > > > > > > > > > > > > > > > > > > > Pulsar connector, we need to >> > have >> > > >> > FLIP-27 >> > > >> > > > by >> > > >> > > > > > late >> > > >> > > > > > > > > Oct. >> > > >> > > > > > > > > > > > There >> > > >> > > > > > > > > > > > > are >> > > >> > > > > > > > > > > > > > > > > still >> > > >> > > > > > > > > > > > > > > > > > 7 >> > > >> > > > > > > > > > > > > > > > > > > > weeks. Personally I think it >> is >> > > >> doable. >> > > >> > > If >> > > >> > > > > > > FLIP-27 >> > > >> > > > > > > > is >> > > >> > > > > > > > > > not >> > > >> > > > > > > > > > > > > ready >> > > >> > > > > > > > > > > > > > > by >> > > >> > > > > > > > > > > > > > > > > late >> > > >> > > > > > > > > > > > > > > > > > > > Oct, we can review and check >> in >> > > >> Pulsar >> > > >> > > > > > connector >> > > >> > > > > > > > with >> > > >> > > > > > > > > > the >> > > >> > > > > > > > > > > > > > > existing >> > > >> > > > > > > > > > > > > > > > > > source >> > > >> > > > > > > > > > > > > > > > > > > > interface. This means we will >> > have >> > > >> > Pulsar >> > > >> > > > > > > connector >> > > >> > > > > > > > > in >> > > >> > > > > > > > > > > > Flink >> > > >> > > > > > > > > > > > > > > 1.10, >> > > >> > > > > > > > > > > > > > > > > > either >> > > >> > > > > > > > > > > > > > > > > > > > with or without FLIP-27. >> > > >> > > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > > Because we are going to have >> > > Pulsar >> > > >> > sink >> > > >> > > > and >> > > >> > > > > > > source >> > > >> > > > > > > > > > > checked >> > > >> > > > > > > > > > > > > in >> > > >> > > > > > > > > > > > > > > > > > > separately, >> > > >> > > > > > > > > > > > > > > > > > > > it might make sense to have >> two >> > > >> FLIPs, >> > > >> > > one >> > > >> > > > > for >> > > >> > > > > > > > Pulsar >> > > >> > > > > > > > > > > sink >> > > >> > > > > > > > > > > > > and >> > > >> > > > > > > > > > > > > > > > > another >> > > >> > > > > > > > > > > > > > > > > > > for >> > > >> > > > > > > > > > > > > > > > > > > > Pulsar source. And we can >> start >> > > the >> > > >> > work >> > > >> > > on >> > > >> > > > > > > Pulsar >> > > >> > > > > > > > > sink >> > > >> > > > > > > > > > > > right >> > > >> > > > > > > > > > > > > > > away. >> > > >> > > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > > Thanks, >> > > >> > > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin >> > > >> > > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 >> PM >> > > Sijie >> > > >> > Guo >> > > >> > > < >> > > >> > > > > > > > > > > > guosi...@gmail.com >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > wrote: >> > > >> > > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > > > Thank you Bowen and Becket. >> > > >> > > > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > > > What's the take from Flink >> > > >> community? >> > > >> > > > Shall >> > > >> > > > > > we >> > > >> > > > > > > > wait >> > > >> > > > > > > > > > for >> > > >> > > > > > > > > > > > > FLIP-27 >> > > >> > > > > > > > > > > > > > > > or >> > > >> > > > > > > > > > > > > > > > > > > shall we >> > > >> > > > > > > > > > > > > > > > > > > > > proceed to next steps? And >> > what >> > > the >> > > >> > > next >> > > >> > > > > > steps >> > > >> > > > > > > > are? >> > > >> > > > > > > > > > :-) >> > > >> > > > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > > > Thanks, >> > > >> > > > > > > > > > > > > > > > > > > > > Sijie >> > > >> > > > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at >> 2:43 PM >> > > >> Bowen >> > > >> > > Li < >> > > >> > > > > > > > > > > > > bowenl...@gmail.com> >> > > >> > > > > > > > > > > > > > > > > wrote: >> > > >> > > > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > > > > Hi, >> > > >> > > > > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > > > > I think having a Pulsar >> > > connector >> > > >> > in >> > > >> > > > > Flink >> > > >> > > > > > > can >> > > >> > > > > > > > > be a >> > > >> > > > > > > > > > > > good >> > > >> > > > > > > > > > > > > > > mutual >> > > >> > > > > > > > > > > > > > > > > > > benefit >> > > >> > > > > > > > > > > > > > > > > > > > > to >> > > >> > > > > > > > > > > > > > > > > > > > > > both communities. >> > > >> > > > > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > > > > Another perspective is >> that >> > > >> Pulsar >> > > >> > > > > > connector >> > > >> > > > > > > is >> > > >> > > > > > > > > the >> > > >> > > > > > > > > > > 1st >> > > >> > > > > > > > > > > > > > > > streaming >> > > >> > > > > > > > > > > > > > > > > > > > > connector >> > > >> > > > > > > > > > > > > > > > > > > > > > that integrates with >> Flink's >> > > >> > metadata >> > > >> > > > > > > > management >> > > >> > > > > > > > > > > system >> > > >> > > > > > > > > > > > > and >> > > >> > > > > > > > > > > > > > > > > Catalog >> > > >> > > > > > > > > > > > > > > > > > > APIs. >> > > >> > > > > > > > > > > > > > > > > > > > > > It'll be cool to see how >> the >> > > >> > > > integration >> > > >> > > > > > > turns >> > > >> > > > > > > > > out >> > > >> > > > > > > > > > > and >> > > >> > > > > > > > > > > > > > > whether >> > > >> > > > > > > > > > > > > > > > we >> > > >> > > > > > > > > > > > > > > > > > > need to >> > > >> > > > > > > > > > > > > > > > > > > > > > improve Flink Catalog >> stack, >> > > >> which >> > > >> > > are >> > > >> > > > > > > > currently >> > > >> > > > > > > > > in >> > > >> > > > > > > > > > > > > Beta, to >> > > >> > > > > > > > > > > > > > > > > cater >> > > >> > > > > > > > > > > > > > > > > > to >> > > >> > > > > > > > > > > > > > > > > > > > > > streaming source/sink. >> Thus >> > > I'm >> > > >> in >> > > >> > > > favor >> > > >> > > > > of >> > > >> > > > > > > > > merging >> > > >> > > > > > > > > > > > > Pulsar >> > > >> > > > > > > > > > > > > > > > > > connector >> > > >> > > > > > > > > > > > > > > > > > > into >> > > >> > > > > > > > > > > > > > > > > > > > > > Flink 1.10. >> > > >> > > > > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > > > > I'd suggest to submit >> > smaller >> > > >> sized >> > > >> > > > PRs, >> > > >> > > > > > e.g. >> > > >> > > > > > > > > maybe >> > > >> > > > > > > > > > > one >> > > >> > > > > > > > > > > > > for >> > > >> > > > > > > > > > > > > > > > basic >> > > >> > > > > > > > > > > > > > > > > > > > > > source/sink >> functionalities >> > > and >> > > >> > > another >> > > >> > > > > for >> > > >> > > > > > > > > schema >> > > >> > > > > > > > > > > and >> > > >> > > > > > > > > > > > > > > catalog >> > > >> > > > > > > > > > > > > > > > > > > > > integration, >> > > >> > > > > > > > > > > > > > > > > > > > > > just to make them easier >> to >> > > >> review. >> > > >> > > > > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > > > > It doesn't seem to hurt >> to >> > > wait >> > > >> for >> > > >> > > > > > FLIP-27. >> > > >> > > > > > > > But >> > > >> > > > > > > > > I >> > > >> > > > > > > > > > > > don't >> > > >> > > > > > > > > > > > > > > think >> > > >> > > > > > > > > > > > > > > > > > > FLIP-27 >> > > >> > > > > > > > > > > > > > > > > > > > > > should be a blocker in >> cases >> > > >> where >> > > >> > it >> > > >> > > > > > cannot >> > > >> > > > > > > > make >> > > >> > > > > > > > > > its >> > > >> > > > > > > > > > > > way >> > > >> > > > > > > > > > > > > > > into >> > > >> > > > > > > > > > > > > > > > > 1.10 >> > > >> > > > > > > > > > > > > > > > > > > or >> > > >> > > > > > > > > > > > > > > > > > > > > > doesn't leave reasonable >> > > amount >> > > >> of >> > > >> > > time >> > > >> > > > > for >> > > >> > > > > > > > > > > committers >> > > >> > > > > > > > > > > > to >> > > >> > > > > > > > > > > > > > > > review >> > > >> > > > > > > > > > > > > > > > > or >> > > >> > > > > > > > > > > > > > > > > > > for >> > > >> > > > > > > > > > > > > > > > > > > > > > Pulsar connector to fully >> > > adapt >> > > >> to >> > > >> > > new >> > > >> > > > > > > > > interfaces. >> > > >> > > > > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > > > > Bowen >> > > >> > > > > > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > > > >
