Thanks, Stephan. Sounds good to me. We can still try our best to get new Pulsar connector in Flink 1.10. In case we do not have time to do that, we will prominently link the Pulsar connector from the Flink connector docs.
Thanks, Jiangjie (Becket) Qin On Mon, Sep 23, 2019 at 4:11 PM Stephan Ewen <se...@apache.org> wrote: > Okay, I see your point, Becket. > > Then let us prominently link the Pulsar connector from the Flink connector > docs then, so that users can find it easily. > > As soon as FLIP 27 is done, we reach out the Pulsar folks to contribute a > new connector. > > On Mon, Sep 23, 2019 at 3:11 AM Becket Qin <becket....@gmail.com> wrote: > > > Hi Stephan, > > > > I have no doubt about the value of adding Pulsar connector to Flink repo. > > My concern is about how exactly we are going to do it. > > > > As mentioned before, I believe that we can handle connectors more > > > pragmatically and less strict than the core of Flink, if it helps > > unlocking > > > users. > > > > I can see the benefit of being less restrict for the initial connector > code > > adoption. However, I don't think we should be less restrict on the > > maintenance commitment once the code is in Flink repo. It only makes > sense > > to check in something and ask users to use if we plan to maintain it. > > > > If I understand correctly, the current plan so far is following: > > 1. release 1.10 > > - Check in Pulsar connector on old interface and label it as beta > > version. > > - encourage users to try it and report bugs. > > 2. release 1.11 > > - Check in Pulsar connector on new interface (a.k.a new Pulsar > > connector) and label it as beta version > > - Deprecate the old Pulsar connector > > - Fix bugs reported on old Pulsar connector from release 1.10 > > - Ask users to migrate from old Pulsar connector to new Pulsar > connector > > 3. release 1.12 > > - Announce end of support for old Pulsar connector and remove the code > > - Fix bugs reported on new Pulsar connector. > > > > If this is the plan, it seems neither Flink nor the users trying the old > > Pulsar connector will benefit from this experimental old Pulsar > connector, > > because whatever feedbacks we got or bugs we fix on the old Pulsar > > connector are immediately thrown away in one or two releases. > > > > If we check in the old Pulsar connector right now, the only option I see > is > > to maintain it for a while (e.g. a year or more). IMO, the immediate > > deprecation and code removal hurts the users much more than asking them > to > > wait for another release. I personally think that we can avoid this > > maintenance burden by going directly to the new Pulsar connector, > > especially given that users can still use the connector even if they are > > not in Flink repo. That said, I am OK with maintaining both old and new > > Pulsar connector if we believe that having the Pulsar connector available > > right now in Flink repo is more important. > > > > Thanks, > > > > Jiangjie (Becket) Qin > > > > On Sun, Sep 22, 2019 at 9:10 PM Stephan Ewen <se...@apache.org> wrote: > > > > > My assumption is as Sijie's, that once the connector is either part of > > > Flink, or part of the streamnative repo. No double maintenance. > > > > > > I feel this discussion is very much caught in problems that are all > > > solvable if we want to solve them. > > > Maybe we can think what our goal for users and the communities is? > > > > > > - Do we want to help build a relationship between the Pulsar and > Flink > > > open source communities? > > > - Will users find a connector in the streamnative repository? > > > - Will users trust a connector that is not part of Flink as much? > > > > > > And then decide what is best according to the overall goals there. > > > As mentioned before, I believe that we can handle connectors more > > > pragmatically and less strict than the core of Flink, if it helps > > unlocking > > > users. > > > > > > Best, > > > Stephan > > > > > > > > > > > > On Fri, Sep 20, 2019 at 2:10 PM Sijie Guo <guosi...@gmail.com> wrote: > > > > > > > Thanks Becket. > > > > > > > > I think it is better for the Flink community to judge the benefits of > > > doing > > > > this. I was trying to provide some views from outsiders. > > > > > > > > Thanks, > > > > Sijie > > > > > > > > On Fri, Sep 20, 2019 at 10:25 AM Becket Qin <becket....@gmail.com> > > > wrote: > > > > > > > > > Hi Sijie, > > > > > > > > > > Yes, we will have to support existing old connectors and new > > connectors > > > > in > > > > > parallel for a while. We have to take that maintenance overhead > > because > > > > > existing connectors have been used by the users for a long time. I > > > guess > > > > It > > > > > may take at least a year for us to fully remove the old connectors. > > > > > > > > > > Process wise, we can do the same for Pulsar connector. But I am not > > > sure > > > > if > > > > > we want to have the same burden on Pulsar connector, and I would > like > > > to > > > > > understand the benefit of doing that. > > > > > > > > > > For users, the benefit of having the old Pulsar connector checked > in > > > > seems > > > > > limited because 1) that code base will be immediately deprecated in > > the > > > > > next release in 3-4 months; 2) users can always use it even if it > is > > > not > > > > in > > > > > the Flink code base. Admittedly it is not as convenient as having > it > > in > > > > > Flink code base, but doesn't seem super either. And after 3-4 > months, > > > > users > > > > > can just use the new connector in Flink repo. > > > > > > > > > > For Flink developers, the old connector code base is not something > > that > > > > we > > > > > want to evolve later. Instead, these code will be deprecated and > > > > > removed. So why do we want to get a beta version out to attract > > people > > > to > > > > > use something we don't want to maintain? > > > > > > > > > > Thanks, > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > On Fri, Sep 20, 2019 at 10:12 AM Sijie Guo <guosi...@gmail.com> > > wrote: > > > > > > > > > > > Thanks everyone here. Sorry for jumping into the discussion here. > > > > > > > > > > > > I am not very familiar about the deprecation process in Flink. > If I > > > > > > misunderstood the process, please fix me. > > > > > > > > > > > > As far as I understand, FLIP-27 is introducing a new unified API > > for > > > > > > connectors. After it introduces the new API > > > > > > and before moving all the existing connectors from old API to new > > > API, > > > > > both > > > > > > old ApI and new API will co-exist > > > > > > for a while until Flink moves all existing connectors to new API. > > So > > > > the > > > > > > Pulsar connector (using old API) can > > > > > > follow the deprecation process with other connector using old API > > and > > > > the > > > > > > deprecation of old API, no? > > > > > > > > > > > > If that's the case, I think contributing the current connector > back > > > to > > > > > > Flink rather than maintaining it outside Flink > > > > > > would provide a bit more benefits. We can deprecate the existing > > > > > > streamnative/pulsar-flink repo and point the users > > > > > > to use the connector in Flink repo. So all the review processes > > will > > > > > happen > > > > > > within Flink for both old connector and > > > > > > new connector. It also reduces the confusions for the users as > the > > > > > > documentation and code base happen in one place. > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > - Sijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 20, 2019 at 12:53 AM Becket Qin < > becket....@gmail.com> > > > > > wrote: > > > > > > > > > > > > > Thanks for the explanation, Stephan. I have a few questions / > > > > thoughts. > > > > > > > > > > > > > > So that means we will remove the old connector without a major > > > > version > > > > > > > bump, is that correct? > > > > > > > > > > > > > > I am not 100% sure if mixing 1.10 connectors with 1.11 > connectors > > > > will > > > > > > > always work because we saw some dependency class collisions in > > the > > > > > past. > > > > > > To > > > > > > > make it safe we may have to maintain the old code for one more > > > > release. > > > > > > > > > > > > > > To be honest I am still wondering if we have to put the old > > > connector > > > > > in > > > > > > > Flink repo. if we check in the old connector to Flink. We will > > end > > > up > > > > > in > > > > > > > the following situation: > > > > > > > 1. Old connector in streamnative/pulsar-flink repo. > > > > > > > 2. Old connector in Flink Repo, which may be different from the > > one > > > > in > > > > > > > Pulsar repo. (Added in 1.10, deprecated in 1.11, removed in > 1.12) > > > > > > > 3. New connector in Flink Repo. > > > > > > > > > > > > > > We need to think about how to make the users in each case > happy. > > > > > > > - For users of (1), I assume Sijie and Yijie will have to > > maintain > > > > the > > > > > > code > > > > > > > a bit longer for its own compatibility even after we have (2). > In > > > > that > > > > > > > case, bugs found in old connector may or may not need to be > fixed > > > in > > > > > both > > > > > > > Flink and the streamnative/pulsar-flink repo. > > > > > > > - For users of (2), will we provide bug fixes? If we do, it > will > > > be a > > > > > > > little awkward because those bug fixes will be immediately > > > deprecated > > > > > in > > > > > > > 1.11, and removed in 1.12. So we are essentially asking users > to > > > > > migrate > > > > > > > away from the bug fix. After Flink 1.12, users may still have > to > > > > switch > > > > > > to > > > > > > > use (3) due to the potential dependency class conflicts > mentioned > > > > > above. > > > > > > > - Users of (3) have a much easier life and don't need to worry > > too > > > > > much. > > > > > > > > > > > > > > The above story seems a little complicated to tell. I think it > > will > > > > be > > > > > > much > > > > > > > easier to not have (2) at all. > > > > > > > 1. Old connector in streamnative/pulsar-flink repo. > > > > > > > 3. New connector in Flink Repo. > > > > > > > > > > > > > > - Old connector will only be maintained in > > > streamnative/pulsar-flink > > > > > repo > > > > > > > until it is fully deprecated. Users can always use the existing > > > > Pulsar > > > > > > > connector in that repo. > > > > > > > - New connector will be in Flink repo and maintained like the > > other > > > > > > > connectors. > > > > > > > > > > > > > > This seems much simpler for users understand and they will not > be > > > > > blocked > > > > > > > from using the old connector. If the concern is about the > quality > > > of > > > > > the > > > > > > > connector in streamnative/pulsar-flink repo, is it enough for > us > > > just > > > > > to > > > > > > > review the code in streamnative/pulsar-flink connector to make > > sure > > > > it > > > > > > > looks good from Flink's perspective? > > > > > > > > > > > > > > What do you think? > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 19, 2019 at 6:58 PM Stephan Ewen <se...@apache.org > > > > > > wrote: > > > > > > > > > > > > > > > My take would be the following: > > > > > > > > > > > > > > > > - If we merge the connector now and replace it with a > FLIP-27 > > > > > version > > > > > > > > before the 1.10 release, then we need no deprecation process > > > > > > > > - If we don't manage to replace it with a FLIP-27 version > > > before > > > > > the > > > > > > > 1.10 > > > > > > > > release, than it is good that we have the other version, so > no > > > > users > > > > > > get > > > > > > > > blocked. > > > > > > > > > > > > > > > > In the latter case we can see how we want to do it. Immediate > > > > removal > > > > > > of > > > > > > > > the old version or deprecation label and keeping it for one > > more > > > > > > release. > > > > > > > > Given that you should be able to use a Flink 1.10 connector > > with > > > > > Flink > > > > > > > 1.11 > > > > > > > > as well (stable public APIs) there is also a workaround if > you > > > need > > > > > an > > > > > > > old > > > > > > > > connector in a newer version. So immediate removal might even > > be > > > > > > > feasible. > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 19, 2019 at 11:09 AM Becket Qin < > > > becket....@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Stephan, > > > > > > > > > > > > > > > > > > Thanks for the clarification. I completely agree with you > and > > > > > Thomas > > > > > > on > > > > > > > > the > > > > > > > > > process of adding connectors to Flink repo. However, I am > > > > wondering > > > > > > > what > > > > > > > > is > > > > > > > > > the deprecation process? Given the main concern here was > that > > > we > > > > > may > > > > > > > have > > > > > > > > > to maintain two Pulsar connector code bases until the old > one > > > is > > > > > > > removed > > > > > > > > > from the repo, it would be good to know how long we have to > > do > > > > > that. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen < > > se...@apache.org > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Some quick thoughts on the connector contribution > process. > > I > > > > > > > basically > > > > > > > > > > reiterate here what Thomas mentioned in another thread > > about > > > > the > > > > > > > > Kinesis > > > > > > > > > > connector. > > > > > > > > > > > > > > > > > > > > For connectors, we should favor a low-overhead > contribution > > > > > > process, > > > > > > > > and > > > > > > > > > > accept user code and changes more readily than in the > core > > > > > system. > > > > > > > > > > That is because connectors have both a big variety of > > > scenarios > > > > > > they > > > > > > > > get > > > > > > > > > > used in (only through use and many small contributions do > > > they > > > > > > become > > > > > > > > > > really useful over time) and at the same time, and > > committers > > > > do > > > > > > not > > > > > > > > use > > > > > > > > > > the connector themselves and usually cannot foresee too > > well > > > > what > > > > > > is > > > > > > > > > > needed. > > > > > > > > > > > > > > > > > > > > Further more, a missing connector (or connector feature) > is > > > > > often a > > > > > > > > > bigger > > > > > > > > > > show stopper for users than a missing API or system > > feature. > > > > > > > > > > > > > > > > > > > > Along these lines of thougt, the conclusion would be to > > take > > > > the > > > > > > > Pulsar > > > > > > > > > > connector now, focus the review on > legal/dependencies/rough > > > > code > > > > > > > style > > > > > > > > > and > > > > > > > > > > conventions, label it as "beta" (in the sense of "new > code" > > > > that > > > > > is > > > > > > > > "not > > > > > > > > > > yet tested through longer use") and go ahead. And then > > evolve > > > > it > > > > > > > > quickly > > > > > > > > > > without putting formal blockers in the way, meaning also > > > > adding a > > > > > > new > > > > > > > > > FLIP > > > > > > > > > > 27 version when it is there. > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Stephan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin < > > > > becket....@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Yijie, > > > > > > > > > > > > > > > > > > > > > > Could you please follow the FLIP process to start a new > > > FLIP > > > > > > > > > [DISCUSSION] > > > > > > > > > > > thread in the mailing list? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process > > > > > > > > > > > > > > > > > > > > > > I see two FLIP-69 discussion in the mailing list now. > So > > > > there > > > > > > is a > > > > > > > > > FLIP > > > > > > > > > > > number collision. Can you change the FLIP number to 72? > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong < > > > > > walter...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi Yijie, > > > > > > > > > > > > > > > > > > > > > > > > Thanks for sharing the pulsar FLIP. > > > > > > > > > > > > Would you mind enabling comments/suggestions on the > > > google > > > > > doc > > > > > > > > link? > > > > > > > > > > This > > > > > > > > > > > > way the contributors from the community can comment > on > > > the > > > > > doc. > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > Rong > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen < > > > > > > > > > henry.yijies...@gmail.com> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > I've drafted a FLIP that describes the current > design > > > of > > > > > the > > > > > > > > Pulsar > > > > > > > > > > > > > connector: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit# > > > > > > > > > > > > > > > > > > > > > > > > > > Please take a look and let me know what you think. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > Yijie > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong < > > > > > > > walter...@gmail.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sorry for joining the discussion late and thanks > > > Yijie > > > > & > > > > > > > Sijie > > > > > > > > > for > > > > > > > > > > > > > driving > > > > > > > > > > > > > > the discussion. > > > > > > > > > > > > > > I also think the Pulsar connector would be a very > > > > > valuable > > > > > > > > > addition > > > > > > > > > > > to > > > > > > > > > > > > > > Flink. I can also help out a bit on the review > side > > > :-) > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regarding the timeline, I also share concerns > with > > > > Becket > > > > > > on > > > > > > > > the > > > > > > > > > > > > > > relationship between the new Pulsar connector and > > > > > FLIP-27. > > > > > > > > > > > > > > There's also another discussion just started by > > > Stephan > > > > > on > > > > > > > > > dropping > > > > > > > > > > > > Kafka > > > > > > > > > > > > > > 9/10 support on next Flink release [1]. Although > > the > > > > > > > situation > > > > > > > > > is > > > > > > > > > > > > > somewhat > > > > > > > > > > > > > > different, and Kafka 9/10 connector has been in > > Flink > > > > for > > > > > > > > almost > > > > > > > > > > 3-4 > > > > > > > > > > > > > years, > > > > > > > > > > > > > > based on the discussion I am not sure if a major > > > > version > > > > > > > > release > > > > > > > > > > is a > > > > > > > > > > > > > > requirement for removing old connector supports. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think there shouldn't be a blocker if we agree > > the > > > > old > > > > > > > > > connector > > > > > > > > > > > will > > > > > > > > > > > > > be > > > > > > > > > > > > > > removed once FLIP-27 based Pulsar connector is > > there. > > > > As > > > > > > > > Stephan > > > > > > > > > > > > stated, > > > > > > > > > > > > > it > > > > > > > > > > > > > > is easier to contribute the source sooner and > > adjust > > > it > > > > > > > later. > > > > > > > > > > > > > > We should also ensure we clearly communicate the > > > > message: > > > > > > for > > > > > > > > > > > example, > > > > > > > > > > > > > > putting an experimental flag on the pre-FLIP27 > > > > connector > > > > > > page > > > > > > > > of > > > > > > > > > > the > > > > > > > > > > > > > > website, documentations, etc. Any other thoughts? > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Rong > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin < > > > > > > > > becket....@gmail.com > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Technically speaking, removing the old > connector > > > code > > > > > is > > > > > > a > > > > > > > > > > > backwards > > > > > > > > > > > > > > > incompatible change which requires a major > > version > > > > > bump, > > > > > > > i.e. > > > > > > > > > > Flink > > > > > > > > > > > > > 2.x. > > > > > > > > > > > > > > > Given that we don't have a clear plan on when > to > > > have > > > > > the > > > > > > > > next > > > > > > > > > > > major > > > > > > > > > > > > > > > version release, it seems unclear how long the > > old > > > > > > > connector > > > > > > > > > code > > > > > > > > > > > > will > > > > > > > > > > > > > be > > > > > > > > > > > > > > > there if we check it in right now. Or will we > > > remove > > > > > the > > > > > > > old > > > > > > > > > > > > connector > > > > > > > > > > > > > > > without a major version bump? In any case, it > > > sounds > > > > > not > > > > > > > > quite > > > > > > > > > > user > > > > > > > > > > > > > > > friendly to the those who might use the old > > Pulsar > > > > > > > > connector. I > > > > > > > > > > am > > > > > > > > > > > > not > > > > > > > > > > > > > sure > > > > > > > > > > > > > > > if it is worth these potential problems in > order > > to > > > > > have > > > > > > > the > > > > > > > > > > Pulsar > > > > > > > > > > > > > source > > > > > > > > > > > > > > > connector checked in one or two months earlier. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen < > > > > > > > > se...@apache.org > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Agreed, if we check in the old code, we > should > > > make > > > > > it > > > > > > > > clear > > > > > > > > > > that > > > > > > > > > > > > it > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > be removed as soon as the FLIP-27 based > version > > > of > > > > > the > > > > > > > > > > connector > > > > > > > > > > > is > > > > > > > > > > > > > > > there. > > > > > > > > > > > > > > > > We should not commit to maintaining the old > > > > version, > > > > > > that > > > > > > > > > would > > > > > > > > > > > be > > > > > > > > > > > > > indeed > > > > > > > > > > > > > > > > too much overhead. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin < > > > > > > > > > > becket....@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Stephan, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the volunteering to help. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes, the overhead would just be review > > > capacity. > > > > In > > > > > > > > fact, I > > > > > > > > > > am > > > > > > > > > > > > not > > > > > > > > > > > > > > > > worrying > > > > > > > > > > > > > > > > > too much about the review capacity. That is > > > just > > > > a > > > > > > one > > > > > > > > time > > > > > > > > > > > cost. > > > > > > > > > > > > > My > > > > > > > > > > > > > > > > > concern is mainly about the long term > burden. > > > > > Assume > > > > > > we > > > > > > > > > have > > > > > > > > > > > new > > > > > > > > > > > > > source > > > > > > > > > > > > > > > > > interface ready in 1.10 with newly added > > Pulsar > > > > > > > > connectors > > > > > > > > > in > > > > > > > > > > > old > > > > > > > > > > > > > > > > > interface. Later on if we migrate Pulsar to > > new > > > > > > source > > > > > > > > > > > interface, > > > > > > > > > > > > > the > > > > > > > > > > > > > > > old > > > > > > > > > > > > > > > > > Pulsar connector might be deprecated almost > > > > > > immediately > > > > > > > > > after > > > > > > > > > > > > > checked > > > > > > > > > > > > > > > in, > > > > > > > > > > > > > > > > > but we may still have to maintain two code > > > bases. > > > > > For > > > > > > > the > > > > > > > > > > > > existing > > > > > > > > > > > > > > > > > connectors, we have to do that anyways. But > > it > > > > > would > > > > > > be > > > > > > > > > good > > > > > > > > > > to > > > > > > > > > > > > > avoid > > > > > > > > > > > > > > > > > introducing a new connector with the same > > > > problem. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan > Ewen > > < > > > > > > > > > > se...@apache.org > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nice to see this lively discussion about > > the > > > > > Pulsar > > > > > > > > > > > connector. > > > > > > > > > > > > > > > > > > Some thoughts on the open questions: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ## Contribute to Flink or maintain as a > > > > community > > > > > > > > package > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Looks like the discussion is more going > > > towards > > > > > > > > > > > contribution. I > > > > > > > > > > > > > think > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > is good, especially if we think that we > > want > > > to > > > > > > > build a > > > > > > > > > > > > similarly > > > > > > > > > > > > > > > deep > > > > > > > > > > > > > > > > > > integration with Pulsar as we have for > > > example > > > > > with > > > > > > > > > Kafka. > > > > > > > > > > > The > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > already looks like a more thorough > > connector > > > > than > > > > > > > many > > > > > > > > > > others > > > > > > > > > > > > we > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > the repository. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > With either a repo split, or the new > build > > > > > system, > > > > > > I > > > > > > > > hope > > > > > > > > > > > that > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > build > > > > > > > > > > > > > > > > > > overhead is not a problem. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ## Committer Support > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Becket offered some help already, I can > > also > > > > > help a > > > > > > > > bit. > > > > > > > > > I > > > > > > > > > > > hope > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > between us, we can cover this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ## Contribute now, or wait for FLIP-27 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As Becket said, FLIP-27 is actually > making > > > some > > > > > > > PoC-ing > > > > > > > > > > > > > progress, but > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > take 2 more months, I would estimate, > > before > > > it > > > > > is > > > > > > > > fully > > > > > > > > > > > > > available. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If we want to be on the safe side with > the > > > > > > > > contribution, > > > > > > > > > we > > > > > > > > > > > > > should > > > > > > > > > > > > > > > > > > contribute the source sooner and adjust > it > > > > later. > > > > > > > That > > > > > > > > > > would > > > > > > > > > > > > also > > > > > > > > > > > > > > > help > > > > > > > > > > > > > > > > us > > > > > > > > > > > > > > > > > > in case things get crazy towards the 1.10 > > > > feature > > > > > > > > freeze > > > > > > > > > > and > > > > > > > > > > > it > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > hard to find time to review the new > > changes. > > > > > > > > > > > > > > > > > > What would be the overhead of > contributing > > > now? > > > > > > Given > > > > > > > > > that > > > > > > > > > > > the > > > > > > > > > > > > > code > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > already there, it looks like it would be > > only > > > > > > review > > > > > > > > > > > capacity, > > > > > > > > > > > > > right? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > Stephan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie > > Shen < > > > > > > > > > > > > > > > henry.yijies...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your attention and the > > promotion > > > > of > > > > > > this > > > > > > > > > work. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We will prepare a FLIP as soon as > > possible > > > > for > > > > > > more > > > > > > > > > > > specific > > > > > > > > > > > > > > > > > discussions. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For FLIP-27, it seems that we have not > > > > reached > > > > > a > > > > > > > > > > consensus. > > > > > > > > > > > > > > > > Therefore, > > > > > > > > > > > > > > > > > > > I will explain all the functionalities > of > > > the > > > > > > > > existing > > > > > > > > > > > > > connector in > > > > > > > > > > > > > > > > > > > the FLIP (including Source, Sink, and > > > > Catalog) > > > > > to > > > > > > > > > > continue > > > > > > > > > > > > our > > > > > > > > > > > > > > > > > > > discussions in FLIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your kind help. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > Yijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket > > Qin > > > < > > > > > > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Sijie, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If we agree that the goal is to have > > > Pulsar > > > > > > > > connector > > > > > > > > > > in > > > > > > > > > > > > > 1.10, > > > > > > > > > > > > > > > how > > > > > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > do the following: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar > connector > > > to > > > > > > Flink > > > > > > > > main > > > > > > > > > > > repo > > > > > > > > > > > > > as it > > > > > > > > > > > > > > > > is a > > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > > public interface to Flink main repo. > > > > > > > > > > > > > > > > > > > > 1. Start to review the Pulsar sink > > right > > > > away > > > > > > as > > > > > > > > > there > > > > > > > > > > is > > > > > > > > > > > > no > > > > > > > > > > > > > > > change > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > sink interface so far. > > > > > > > > > > > > > > > > > > > > 2. Wait a little bit on FLIP-27. > Flink > > > 1.10 > > > > > is > > > > > > > > going > > > > > > > > > to > > > > > > > > > > > be > > > > > > > > > > > > > code > > > > > > > > > > > > > > > > > freeze > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > late Nov and let's say we give a > month > > to > > > > the > > > > > > > > > > development > > > > > > > > > > > > and > > > > > > > > > > > > > > > > review > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > Pulsar connector, we need to have > > FLIP-27 > > > > by > > > > > > late > > > > > > > > > Oct. > > > > > > > > > > > > There > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > still > > > > > > > > > > > > > > > > > > 7 > > > > > > > > > > > > > > > > > > > > weeks. Personally I think it is > doable. > > > If > > > > > > > FLIP-27 > > > > > > > > is > > > > > > > > > > not > > > > > > > > > > > > > ready > > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > > late > > > > > > > > > > > > > > > > > > > > Oct, we can review and check in > Pulsar > > > > > > connector > > > > > > > > with > > > > > > > > > > the > > > > > > > > > > > > > > > existing > > > > > > > > > > > > > > > > > > source > > > > > > > > > > > > > > > > > > > > interface. This means we will have > > Pulsar > > > > > > > connector > > > > > > > > > in > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > 1.10, > > > > > > > > > > > > > > > > > > either > > > > > > > > > > > > > > > > > > > > with or without FLIP-27. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Because we are going to have Pulsar > > sink > > > > and > > > > > > > source > > > > > > > > > > > checked > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > separately, > > > > > > > > > > > > > > > > > > > > it might make sense to have two > FLIPs, > > > one > > > > > for > > > > > > > > Pulsar > > > > > > > > > > > sink > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > Pulsar source. And we can start the > > work > > > on > > > > > > > Pulsar > > > > > > > > > sink > > > > > > > > > > > > right > > > > > > > > > > > > > > > away. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie > > Guo > > > < > > > > > > > > > > > > guosi...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you Bowen and Becket. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What's the take from Flink > community? > > > > Shall > > > > > > we > > > > > > > > wait > > > > > > > > > > for > > > > > > > > > > > > > FLIP-27 > > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > > shall we > > > > > > > > > > > > > > > > > > > > > proceed to next steps? And what the > > > next > > > > > > steps > > > > > > > > are? > > > > > > > > > > :-) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > Sijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM > Bowen > > > Li < > > > > > > > > > > > > > bowenl...@gmail.com> > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think having a Pulsar connector > > in > > > > > Flink > > > > > > > can > > > > > > > > > be a > > > > > > > > > > > > good > > > > > > > > > > > > > > > mutual > > > > > > > > > > > > > > > > > > > benefit > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > both communities. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Another perspective is that > Pulsar > > > > > > connector > > > > > > > is > > > > > > > > > the > > > > > > > > > > > 1st > > > > > > > > > > > > > > > > streaming > > > > > > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > > > > that integrates with Flink's > > metadata > > > > > > > > management > > > > > > > > > > > system > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > Catalog > > > > > > > > > > > > > > > > > > > APIs. > > > > > > > > > > > > > > > > > > > > > > It'll be cool to see how the > > > > integration > > > > > > > turns > > > > > > > > > out > > > > > > > > > > > and > > > > > > > > > > > > > > > whether > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > need to > > > > > > > > > > > > > > > > > > > > > > improve Flink Catalog stack, > which > > > are > > > > > > > > currently > > > > > > > > > in > > > > > > > > > > > > > Beta, to > > > > > > > > > > > > > > > > > cater > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > streaming source/sink. Thus I'm > in > > > > favor > > > > > of > > > > > > > > > merging > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > > > > > > Flink 1.10. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'd suggest to submit smaller > sized > > > > PRs, > > > > > > e.g. > > > > > > > > > maybe > > > > > > > > > > > one > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > basic > > > > > > > > > > > > > > > > > > > > > > source/sink functionalities and > > > another > > > > > for > > > > > > > > > schema > > > > > > > > > > > and > > > > > > > > > > > > > > > catalog > > > > > > > > > > > > > > > > > > > > > integration, > > > > > > > > > > > > > > > > > > > > > > just to make them easier to > review. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It doesn't seem to hurt to wait > for > > > > > > FLIP-27. > > > > > > > > But > > > > > > > > > I > > > > > > > > > > > > don't > > > > > > > > > > > > > > > think > > > > > > > > > > > > > > > > > > > FLIP-27 > > > > > > > > > > > > > > > > > > > > > > should be a blocker in cases > where > > it > > > > > > cannot > > > > > > > > make > > > > > > > > > > its > > > > > > > > > > > > way > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > 1.10 > > > > > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > > > > > doesn't leave reasonable amount > of > > > time > > > > > for > > > > > > > > > > > committers > > > > > > > > > > > > to > > > > > > > > > > > > > > > > review > > > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > Pulsar connector to fully adapt > to > > > new > > > > > > > > > interfaces. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Bowen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM > > Becket > > > > > Qin < > > > > > > > > > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > You are right. It all depends > on > > > when > > > > > the > > > > > > > new > > > > > > > > > > > source > > > > > > > > > > > > > > > > interface > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > going > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > be ready. Personally I think it > > > would > > > > > be > > > > > > > > there > > > > > > > > > in > > > > > > > > > > > > > about a > > > > > > > > > > > > > > > > month > > > > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > > so. > > > > > > > > > > > > > > > > > > > > > > But > > > > > > > > > > > > > > > > > > > > > > > I could be too optimistic. It > > would > > > > > also > > > > > > be > > > > > > > > > good > > > > > > > > > > to > > > > > > > > > > > > > hear > > > > > > > > > > > > > > > what > > > > > > > > > > > > > > > > > do > > > > > > > > > > > > > > > > > > > > > Aljoscha > > > > > > > > > > > > > > > > > > > > > > > and Stephan think as they are > > also > > > > > > involved > > > > > > > > in > > > > > > > > > > > > FLIP-27. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In general I think we should > have > > > > > Pulsar > > > > > > > > > > connector > > > > > > > > > > > in > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > 1.10, > > > > > > > > > > > > > > > > > > > > > > > preferably with the new source > > > > > interface. > > > > > > > We > > > > > > > > > can > > > > > > > > > > > also > > > > > > > > > > > > > check > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > right > > > > > > > > > > > > > > > > > > > > > > now > > > > > > > > > > > > > > > > > > > > > > > with old source interface, but > I > > > > > suspect > > > > > > > few > > > > > > > > > > users > > > > > > > > > > > > > will use > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > before > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > next official release. > Therefore, > > > it > > > > > > seems > > > > > > > > > > > reasonable > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > wait a > > > > > > > > > > > > > > > > > > > little > > > > > > > > > > > > > > > > > > > > > > bit > > > > > > > > > > > > > > > > > > > > > > > to see whether we can jump to > the > > > new > > > > > > > source > > > > > > > > > > > > > interface. As > > > > > > > > > > > > > > > > long > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > > > make > > > > > > > > > > > > > > > > > > > > > > > sure Flink 1.10 has it, > waiting a > > > > > little > > > > > > > bit > > > > > > > > > > > doesn't > > > > > > > > > > > > > seem > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > hurt > > > > > > > > > > > > > > > > > > > much. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM > > Till > > > > > > > Rohrmann > > > > > > > > < > > > > > > > > > > > > > > > > > > trohrm...@apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm wondering what the > problem > > > > would > > > > > be > > > > > > > if > > > > > > > > we > > > > > > > > > > > > > committed > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > connector before the new > source > > > > > > interface > > > > > > > > is > > > > > > > > > > > ready. > > > > > > > > > > > > > If I > > > > > > > > > > > > > > > > > > > understood > > > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > > > correctly, then we need to > > > support > > > > > the > > > > > > > old > > > > > > > > > > source > > > > > > > > > > > > > > > interface > > > > > > > > > > > > > > > > > > > anyway > > > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > existing connectors. By > > checking > > > it > > > > > in > > > > > > > > early > > > > > > > > > I > > > > > > > > > > > > could > > > > > > > > > > > > > see > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > benefit > > > > > > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > > > > > > our users could start using > the > > > > > > connector > > > > > > > > > > > earlier. > > > > > > > > > > > > > > > > Moreover, > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > > > > > > > > prevent that the Pulsar > > > integration > > > > > is > > > > > > > > being > > > > > > > > > > > > delayed > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > case > > > > > > > > > > > > > > > > > > > that the > > > > > > > > > > > > > > > > > > > > > > > > source interface should be > > > delayed. > > > > > The > > > > > > > > only > > > > > > > > > > > > > downside I > > > > > > > > > > > > > > > see > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > extra > > > > > > > > > > > > > > > > > > > > > > > > review effort and potential > > fixes > > > > > which > > > > > > > > might > > > > > > > > > > be > > > > > > > > > > > > > > > irrelevant > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > > > > > > source interface > > implementation. > > > I > > > > > > guess > > > > > > > it > > > > > > > > > > > mainly > > > > > > > > > > > > > > > depends > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > how > > > > > > > > > > > > > > > > > > > > > > certain > > > > > > > > > > > > > > > > > > > > > > > > we are when the new source > > > > interface > > > > > > will > > > > > > > > be > > > > > > > > > > > ready. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 > AM > > > > Becket > > > > > > > Qin < > > > > > > > > > > > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Sijie and Yijie, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for sharing your > > > thoughts. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Just want to have some > update > > > on > > > > > > > FLIP-27. > > > > > > > > > > > > Although > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > FLIP > > > > > > > > > > > > > > > > > > > wiki > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > discussion thread has been > > > quiet > > > > > for > > > > > > > some > > > > > > > > > > > time, a > > > > > > > > > > > > > few > > > > > > > > > > > > > > > > > > > committer / > > > > > > > > > > > > > > > > > > > > > > > > > contributors in Flink > > community > > > > > were > > > > > > > > > actually > > > > > > > > > > > > > > > prototyping > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > entire > > > > > > > > > > > > > > > > > > > > > > > > thing. > > > > > > > > > > > > > > > > > > > > > > > > > We have made some good > > progress > > > > > there > > > > > > > but > > > > > > > > > > want > > > > > > > > > > > to > > > > > > > > > > > > > > > update > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > FLIP > > > > > > > > > > > > > > > > > > > > > > wiki > > > > > > > > > > > > > > > > > > > > > > > > > after the entire thing is > > > > verified > > > > > to > > > > > > > > work > > > > > > > > > in > > > > > > > > > > > > case > > > > > > > > > > > > > > > there > > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > > > > > last > > > > > > > > > > > > > > > > > > > > > > > > > minute surprise in the > > > > > > implementation. > > > > > > > I > > > > > > > > > > don't > > > > > > > > > > > > > have an > > > > > > > > > > > > > > > > > exact > > > > > > > > > > > > > > > > > > > ETA > > > > > > > > > > > > > > > > > > > > > yet, > > > > > > > > > > > > > > > > > > > > > > > > but I > > > > > > > > > > > > > > > > > > > > > > > > > guess it is going to be > > within > > > a > > > > > > month > > > > > > > or > > > > > > > > > so. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am happy to review the > > > current > > > > > > Flink > > > > > > > > > Pulsar > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > see if > > > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > > > > > > > > > fit in FLIP-27. It would be > > > good > > > > to > > > > > > > avoid > > > > > > > > > the > > > > > > > > > > > > case > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > checked > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector with some > > > review > > > > > > > efforts > > > > > > > > > and > > > > > > > > > > > > > shortly > > > > > > > > > > > > > > > > after > > > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > > > > > > > Source interface is ready. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 > > AM > > > > > Yijie > > > > > > > > Shen < > > > > > > > > > > > > > > > > > > > > > henry.yijies...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for all the > feedback > > > and > > > > > > > > > > suggestions! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As Sijie said, the goal > of > > > the > > > > > > > > connector > > > > > > > > > > has > > > > > > > > > > > > > always > > > > > > > > > > > > > > > > been > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > provide > > > > > > > > > > > > > > > > > > > > > > > > > > users with the latest > > > features > > > > of > > > > > > > both > > > > > > > > > > > systems > > > > > > > > > > > > as > > > > > > > > > > > > > > > soon > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > possible. > > > > > > > > > > > > > > > > > > > > > > > We > > > > > > > > > > > > > > > > > > > > > > > > > > propose to contribute the > > > > > connector > > > > > > > to > > > > > > > > > > Flink > > > > > > > > > > > > and > > > > > > > > > > > > > hope > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > get > > > > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > > > > > > > suggestions and feedback > > from > > > > > Flink > > > > > > > > > experts > > > > > > > > > > > to > > > > > > > > > > > > > ensure > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > high > > > > > > > > > > > > > > > > > > > > > > > quality > > > > > > > > > > > > > > > > > > > > > > > > > > of the connector. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For FLIP-27, we noticed > its > > > > > > existence > > > > > > > > at > > > > > > > > > > the > > > > > > > > > > > > > > > beginning > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > reworking > > > > > > > > > > > > > > > > > > > > > > > > > > the connector > > implementation > > > > > based > > > > > > on > > > > > > > > > Flink > > > > > > > > > > > > 1.9; > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > also > > > > > > > > > > > > > > > > > > > wanted > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > > build a connector that > > > supports > > > > > > both > > > > > > > > > batch > > > > > > > > > > > and > > > > > > > > > > > > > stream > > > > > > > > > > > > > > > > > > > computing > > > > > > > > > > > > > > > > > > > > > > based > > > > > > > > > > > > > > > > > > > > > > > > > > on it. > > > > > > > > > > > > > > > > > > > > > > > > > > However, it has been > > inactive > > > > for > > > > > > > some > > > > > > > > > > time, > > > > > > > > > > > so > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > decided > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > provide > > > > > > > > > > > > > > > > > > > > > > > > > > a connector with most of > > the > > > > new > > > > > > > > > features, > > > > > > > > > > > such > > > > > > > > > > > > > as > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > type > > > > > > > > > > > > > > > > > > > > > > > system > > > > > > > > > > > > > > > > > > > > > > > > > > and the new catalog API > > > first. > > > > We > > > > > > > will > > > > > > > > > pay > > > > > > > > > > > > > attention > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > progress > > > > > > > > > > > > > > > > > > > > > > > > > > of FLIP-27 continually > and > > > > > > > incorporate > > > > > > > > it > > > > > > > > > > > with > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > > soon > > > > > > > > > > > > > > > > > > > > > > > > > > as possible. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regarding the test status > > of > > > > the > > > > > > > > > connector, > > > > > > > > > > > we > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > following > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > other > > > > > > > > > > > > > > > > > > > > > > > > > > connectors' test in Flink > > > > > > repository > > > > > > > > and > > > > > > > > > > > aimed > > > > > > > > > > > > to > > > > > > > > > > > > > > > > provide > > > > > > > > > > > > > > > > > > > > > > throughout > > > > > > > > > > > > > > > > > > > > > > > > > > tests as we could. We are > > > also > > > > > > happy > > > > > > > to > > > > > > > > > > hear > > > > > > > > > > > > > > > > suggestions > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > supervision from the > Flink > > > > > > community > > > > > > > to > > > > > > > > > > > improve > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > stability and > > > > > > > > > > > > > > > > > > > > > > > > > > performance of the > > connector > > > > > > > > > continuously. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > > > > > > Yijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at > 5:59 > > > AM > > > > > > Sijie > > > > > > > > Guo > > > > > > > > > < > > > > > > > > > > > > > > > > > > guosi...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks everyone for the > > > > > comments > > > > > > > and > > > > > > > > > > > > feedback. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It seems to me that the > > > main > > > > > > > question > > > > > > > > > > here > > > > > > > > > > > is > > > > > > > > > > > > > > > about - > > > > > > > > > > > > > > > > > > "how > > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > > > community maintain the > > > > > > connector?". > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Here are two thoughts > > from > > > > > > myself. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1) I think how and > where > > to > > > > > host > > > > > > > this > > > > > > > > > > > > > integration > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > kind > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > less > > > > > > > > > > > > > > > > > > > > > > > > > > important > > > > > > > > > > > > > > > > > > > > > > > > > > > here. I believe there > can > > > be > > > > > many > > > > > > > > ways > > > > > > > > > to > > > > > > > > > > > > > achieve > > > > > > > > > > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > > > > As part of the > > > contribution, > > > > > what > > > > > > > we > > > > > > > > > are > > > > > > > > > > > > > looking > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > here > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > how > > > > > > > > > > > > > > > > > > > > > > > > these > > > > > > > > > > > > > > > > > > > > > > > > > > two > > > > > > > > > > > > > > > > > > > > > > > > > > > communities can build > the > > > > > > > > collaboration > > > > > > > > > > > > > > > relationship > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > > > > developing > > > > > > > > > > > > > > > > > > > > > > > > > > > the integration between > > > > Pulsar > > > > > > and > > > > > > > > > Flink. > > > > > > > > > > > > Even > > > > > > > > > > > > > we > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > try > > > > > > > > > > > > > > > > > > > our > > > > > > > > > > > > > > > > > > > > > > best > > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > > catch > > > > > > > > > > > > > > > > > > > > > > > > > > > up all the updates in > > Flink > > > > > > > > community. > > > > > > > > > We > > > > > > > > > > > are > > > > > > > > > > > > > still > > > > > > > > > > > > > > > > > > > > > > > > > > > facing the fact that we > > > have > > > > > less > > > > > > > > > > > experiences > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > than > > > > > > > > > > > > > > > > > > > > > folks > > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > > > community. In order to > > make > > > > > sure > > > > > > we > > > > > > > > > > > maintain > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > deliver > > > > > > > > > > > > > > > > > > > > > > > > > > > a high-quality > > pulsar-flink > > > > > > > > integration > > > > > > > > > > to > > > > > > > > > > > > the > > > > > > > > > > > > > > > users > > > > > > > > > > > > > > > > > who > > > > > > > > > > > > > > > > > > > use > > > > > > > > > > > > > > > > > > > > > both > > > > > > > > > > > > > > > > > > > > > > > > > > > technologies, we need > > some > > > > help > > > > > > > from > > > > > > > > > the > > > > > > > > > > > > > experts > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > community. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2) We have been > following > > > > > FLIP-27 > > > > > > > > for a > > > > > > > > > > > > while. > > > > > > > > > > > > > > > > > Originally > > > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > > were > > > > > > > > > > > > > > > > > > > > > > > > > > thinking > > > > > > > > > > > > > > > > > > > > > > > > > > > of contributing the > > > > connectors > > > > > > back > > > > > > > > > after > > > > > > > > > > > > > > > integrating > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > new API introduced in > > > > FLIP-27. > > > > > > But > > > > > > > we > > > > > > > > > > > decided > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > initiate > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > conversation > > > > > > > > > > > > > > > > > > > > > > > > > > > as early as possible. > > > Because > > > > > we > > > > > > > > > believe > > > > > > > > > > > > there > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > > benefits > > > > > > > > > > > > > > > > > > > > > > > > doing > > > > > > > > > > > > > > > > > > > > > > > > > > > it now rather than > later. > > > As > > > > > part > > > > > > > of > > > > > > > > > > > > > contribution, > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > help > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > > > community understand > more > > > > about > > > > > > > > Pulsar > > > > > > > > > > and > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > potential > > > > > > > > > > > > > > > > > > > > > > > integration > > > > > > > > > > > > > > > > > > > > > > > > > > points. > > > > > > > > > > > > > > > > > > > > > > > > > > > Also we can also help > > Flink > > > > > > > community > > > > > > > > > > > verify > > > > > > > > > > > > > the > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > > > > API > > > > > > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > > > > > > well > > > > > > > > > > > > > > > > > > > > > > > > > > > as other new API (e.g. > > > > catalog > > > > > > > API). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > Sijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at > > 5:24 > > > > AM > > > > > > > Becket > > > > > > > > > > Qin < > > > > > > > > > > > > > > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yijie, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the > interest > > > in > > > > > > > > > contributing > > > > > > > > > > > the > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > connector. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In general, I think > > > having > > > > > > Pulsar > > > > > > > > > > > connector > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > strong > > > > > > > > > > > > > > > > > > > > > support > > > > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > > > > > > > > valuable addition to > > > Flink. > > > > > So > > > > > > I > > > > > > > am > > > > > > > > > > happy > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > shepherd > > > > > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > > > > > > effort. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Meanwhile, I would > also > > > > like > > > > > to > > > > > > > > > provide > > > > > > > > > > > > some > > > > > > > > > > > > > > > > context > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > recent > > > > > > > > > > > > > > > > > > > > > > > > > > efforts on > > > > > > > > > > > > > > > > > > > > > > > > > > > > the Flink connectors > > > > > ecosystem. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The current way Flink > > > > > maintains > > > > > > > its > > > > > > > > > > > > > connector has > > > > > > > > > > > > > > > > hit > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > scalability > > > > > > > > > > > > > > > > > > > > > > > > > > bar. > > > > > > > > > > > > > > > > > > > > > > > > > > > > With more and more > > > > connectors > > > > > > > > coming > > > > > > > > > > into > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > repo, > > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > > > > > > facing a > > > > > > > > > > > > > > > > > > > > > > > > > > few > > > > > > > > > > > > > > > > > > > > > > > > > > > > problems such as long > > > build > > > > > and > > > > > > > > > testing > > > > > > > > > > > > > time. To > > > > > > > > > > > > > > > > > > address > > > > > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > > > > > > > > problem, > > > > > > > > > > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > > > > > > > > > have attempted to do > > the > > > > > > > following: > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Split out the > > > connectors > > > > > > into > > > > > > > a > > > > > > > > > > > separate > > > > > > > > > > > > > > > > > repository. > > > > > > > > > > > > > > > > > > > This > > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > > > > temporarily > > > > > > > > > > > > > > > > > > > > > > > > > > > > on hold due to > > potential > > > > > > solution > > > > > > > > to > > > > > > > > > > > > shorten > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > build > > > > > > > > > > > > > > > > > > > time. > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Encourage the > > > connectors > > > > > to > > > > > > > stay > > > > > > > > > as > > > > > > > > > > > > > ecosystem > > > > > > > > > > > > > > > > > > project > > > > > > > > > > > > > > > > > > > > > while > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > > tries > > > > > > > > > > > > > > > > > > > > > > > > > > > > to provide good > support > > > for > > > > > > > > > > functionality > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > compatibility > > > > > > > > > > > > > > > > > > > > > > > tests. > > > > > > > > > > > > > > > > > > > > > > > > > > Robert > > > > > > > > > > > > > > > > > > > > > > > > > > > > has driven to create > a > > > > Flink > > > > > > > > > Ecosystem > > > > > > > > > > > > > project > > > > > > > > > > > > > > > > > website > > > > > > > > > > > > > > > > > > > and it > > > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > > > going > > > > > > > > > > > > > > > > > > > > > > > > > > > > through some final > > > approval > > > > > > > > process. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Given the above > > efforts, > > > it > > > > > > would > > > > > > > > be > > > > > > > > > > > great > > > > > > > > > > > > to > > > > > > > > > > > > > > > first > > > > > > > > > > > > > > > > > see > > > > > > > > > > > > > > > > > > > if we > > > > > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector as > an > > > > > > ecosystem > > > > > > > > > > project > > > > > > > > > > > > with > > > > > > > > > > > > > > > great > > > > > > > > > > > > > > > > > > > support. > > > > > > > > > > > > > > > > > > > > > It > > > > > > > > > > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > > > > > > > good to hear how the > > > Flink > > > > > > Pulsar > > > > > > > > > > > connector > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > tested > > > > > > > > > > > > > > > > > > > > > currently > > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > see > > > > > > > > > > > > > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > > > > > > > > > > > > > we can learn > something > > to > > > > > > > maintain > > > > > > > > it > > > > > > > > > > as > > > > > > > > > > > an > > > > > > > > > > > > > > > > ecosystem > > > > > > > > > > > > > > > > > > > project > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > > good > > > > > > > > > > > > > > > > > > > > > > > > > > > > quality and test > > > coverage. > > > > If > > > > > > the > > > > > > > > > > quality > > > > > > > > > > > > as > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > ecosystem > > > > > > > > > > > > > > > > > > > > > > project > > > > > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > > > > hard > > > > > > > > > > > > > > > > > > > > > > > > > > > > to guarantee, we may > as > > > > well > > > > > > > adopt > > > > > > > > it > > > > > > > > > > > into > > > > > > > > > > > > > the > > > > > > > > > > > > > > > main > > > > > > > > > > > > > > > > > > repo. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > BTW, another ongoing > > > effort > > > > > is > > > > > > > > > FLIP-27 > > > > > > > > > > > > where > > > > > > > > > > > > > we > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > making > > > > > > > > > > > > > > > > > > > > > > > changes > > > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink source > connector > > > > > > > architecture > > > > > > > > > and > > > > > > > > > > > > > > > interface. > > > > > > > > > > > > > > > > > This > > > > > > > > > > > > > > > > > > > > > change > > > > > > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > > > > > > > > likely > > > > > > > > > > > > > > > > > > > > > > > > > > > > land in 1.10. > Therefore > > > > > timing > > > > > > > > wise, > > > > > > > > > if > > > > > > > > > > > we > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > going > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > > > > > connector in main > > repo, I > > > > am > > > > > > > > > wondering > > > > > > > > > > if > > > > > > > > > > > > we > > > > > > > > > > > > > > > should > > > > > > > > > > > > > > > > > > hold > > > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > > little > > > > > > > > > > > > > > > > > > > > > > > > bit > > > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > > > let the Pulsar > > connector > > > > > adapt > > > > > > to > > > > > > > > the > > > > > > > > > > new > > > > > > > > > > > > > > > interface > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > avoid > > > > > > > > > > > > > > > > > > > > > > > > shortly > > > > > > > > > > > > > > > > > > > > > > > > > > > > deprecated work? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 > at > > > 4:32 > > > > > PM > > > > > > > > > Chesnay > > > > > > > > > > > > > Schepler < > > > > > > > > > > > > > > > > > > > > > > > > ches...@apache.org> > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm quite worried > > that > > > we > > > > > may > > > > > > > end > > > > > > > > > up > > > > > > > > > > > > > repeating > > > > > > > > > > > > > > > > > > history. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There were already > 2 > > > > > attempts > > > > > > > at > > > > > > > > > > > > > contributing a > > > > > > > > > > > > > > > > > > pulsar > > > > > > > > > > > > > > > > > > > > > > > connector, > > > > > > > > > > > > > > > > > > > > > > > > > > both > > > > > > > > > > > > > > > > > > > > > > > > > > > > > of which failed > > because > > > > no > > > > > > > > > committer > > > > > > > > > > > was > > > > > > > > > > > > > > > getting > > > > > > > > > > > > > > > > > > > involved, > > > > > > > > > > > > > > > > > > > > > > > > despite > > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > contributor > opening a > > > > > > dedicated > > > > > > > > > > > > discussion > > > > > > > > > > > > > > > thread > > > > > > > > > > > > > > > > > > > about the > > > > > > > > > > > > > > > > > > > > > > > > > > contribution > > > > > > > > > > > > > > > > > > > > > > > > > > > > > beforehand and > > getting > > > > > > several > > > > > > > > +1's > > > > > > > > > > > from > > > > > > > > > > > > > > > > > committers. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We should really > make > > > > sure > > > > > > that > > > > > > > > if > > > > > > > > > we > > > > > > > > > > > > > > > > > welcome/approve > > > > > > > > > > > > > > > > > > > such > > > > > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > > > > > > > > > contribution it > will > > > > > actually > > > > > > > get > > > > > > > > > the > > > > > > > > > > > > > attention > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > deserves. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As such, I'm > inclined > > > to > > > > > > > > recommend > > > > > > > > > > > > > maintaining > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > > > > > > > outside > > > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink. We could > link > > to > > > > it > > > > > > from > > > > > > > > the > > > > > > > > > > > > > > > documentation > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > give > > > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > > > > > > > > > exposure. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > With the upcoming > > page > > > > for > > > > > > > > sharing > > > > > > > > > > > > > artifacts > > > > > > > > > > > > > > > > among > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > community > > > > > > > > > > > > > > > > > > > > > > > > > > (what's > > > > > > > > > > > > > > > > > > > > > > > > > > > > > the state of that > > > > anyway?), > > > > > > > this > > > > > > > > > may > > > > > > > > > > > be a > > > > > > > > > > > > > > > better > > > > > > > > > > > > > > > > > > > option. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 04/09/2019 > 10:16, > > > Till > > > > > > > > Rohrmann > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks a lot for > > > > starting > > > > > > > this > > > > > > > > > > > > discussion > > > > > > > > > > > > > > > > Yijie. > > > > > > > > > > > > > > > > > I > > > > > > > > > > > > > > > > > > > think > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > connector would > be > > a > > > > very > > > > > > > > > valuable > > > > > > > > > > > > > addition > > > > > > > > > > > > > > > > since > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > becomes > > > > > > > > > > > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > more popular and > it > > > > would > > > > > > > > further > > > > > > > > > > > > expand > > > > > > > > > > > > > > > > Flink's > > > > > > > > > > > > > > > > > > > > > > > > > interoperability. > > > > > > > > > > > > > > > > > > > > > > > > > > Also > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > from a project > > > > > perspective > > > > > > it > > > > > > > > > makes > > > > > > > > > > > > > sense for > > > > > > > > > > > > > > > > me > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > place > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > the downstream > > > project. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My main > > > > concern/question > > > > > is > > > > > > > how > > > > > > > > > can > > > > > > > > > > > the > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > community > > > > > > > > > > > > > > > > > > > > > > > > maintain > > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > connector? We > have > > > seen > > > > > in > > > > > > > the > > > > > > > > > past > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > connectors > > > > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > most > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > actively > developed > > > > > > components > > > > > > > > > > because > > > > > > > > > > > > > they > > > > > > > > > > > > > > > need > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > kept > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > sync > > > > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > external system > and > > > > with > > > > > > > Flink. > > > > > > > > > > Given > > > > > > > > > > > > > that > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > community > > > > > > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > > > > > > > willing > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > to help with > > > > maintaining, > > > > > > > > > improving > > > > > > > > > > > and > > > > > > > > > > > > > > > > evolving > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > connector, > > > > > > > > > > > > > > > > > > > > > > > > > I'm > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > optimistic that > we > > > can > > > > > > > achieve > > > > > > > > > > this. > > > > > > > > > > > > > Hence, > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > > contributing > > > > > > > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > > > > > back > > > > > > > > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, > 2019 > > > at > > > > > 2:03 > > > > > > > AM > > > > > > > > > > Sijie > > > > > > > > > > > > Guo > > > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > guosi...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi Yun, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Since I was the > > main > > > > > > driver > > > > > > > > > behind > > > > > > > > > > > > > > > FLINK-9641 > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > FLINK-9168, > > > > > > > > > > > > > > > > > > > > > > > > > let > > > > > > > > > > > > > > > > > > > > > > > > > > me > > > > > > > > > > > > > > > > > > > > > > > > > > > > > try to > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> add more context > > on > > > > > this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and > > > > > FLINK-9168 > > > > > > > was > > > > > > > > > > > created > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > bringing > > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > > > > > > source > > > > > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> sink for Flink. > > The > > > > > > > > integration > > > > > > > > > > was > > > > > > > > > > > > done > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > 1.6.0. > > > > > > > > > > > > > > > > > > > > > > > We > > > > > > > > > > > > > > > > > > > > > > > > > > sent out > > > > > > > > > > > > > > > > > > > > > > > > > > > > > pull > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> requests about a > > > year > > > > > ago > > > > > > > and > > > > > > > > we > > > > > > > > > > > ended > > > > > > > > > > > > > up > > > > > > > > > > > > > > > > > > > maintaining > > > > > > > > > > > > > > > > > > > > > > those > > > > > > > > > > > > > > > > > > > > > > > > > > connectors > > > > > > > > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar for > Pulsar > > > > users > > > > > to > > > > > > > use > > > > > > > > > > Flink > > > > > > > > > > > > to > > > > > > > > > > > > > > > > process > > > > > > > > > > > > > > > > > > > event > > > > > > > > > > > > > > > > > > > > > > > streams > > > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> (See > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink > > > > > > > > > > > > > > > > > > > > > > > > > ). > > > > > > > > > > > > > > > > > > > > > > > > > > The > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> 1.6 integration > is > > > > > pretty > > > > > > > > simple > > > > > > > > > > and > > > > > > > > > > > > > there > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > no > > > > > > > > > > > > > > > > > > > schema > > > > > > > > > > > > > > > > > > > > > > > > > > > > considerations. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> In the past > year, > > we > > > > > have > > > > > > > > made a > > > > > > > > > > lot > > > > > > > > > > > > of > > > > > > > > > > > > > > > > changes > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > brought > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar schema as > > the > > > > > > > > first-class > > > > > > > > > > > > > citizen in > > > > > > > > > > > > > > > > > > Pulsar. > > > > > > > > > > > > > > > > > > > We > > > > > > > > > > > > > > > > > > > > > > also > > > > > > > > > > > > > > > > > > > > > > > > > > integrated > > > > > > > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> other computing > > > > engines > > > > > > for > > > > > > > > > > > processing > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > event > > > > > > > > > > > > > > > > > > > > > > streams > > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> schema. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> It led us to > > rethink > > > > how > > > > > > to > > > > > > > > > > > integrate > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > best > > > > > > > > > > > > > > > > > > > > > > > > > way. > > > > > > > > > > > > > > > > > > > > > > > > > > Then > > > > > > > > > > > > > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> reimplement the > > > > > > pulsar-flink > > > > > > > > > > > > connectors > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > ground > > > > > > > > > > > > > > > > > > > > > up > > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > > > schema > > > > > > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> bring table API > > and > > > > > > catalog > > > > > > > > API > > > > > > > > > as > > > > > > > > > > > the > > > > > > > > > > > > > > > > > first-class > > > > > > > > > > > > > > > > > > > > > citizen > > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> integration. > With > > > that > > > > > > being > > > > > > > > > said, > > > > > > > > > > > in > > > > > > > > > > > > > the > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > > > pulsar-flink > > > > > > > > > > > > > > > > > > > > > > > > > > > > > implementation, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> you can register > > > > pulsar > > > > > > as a > > > > > > > > > flink > > > > > > > > > > > > > catalog > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > query / > > > > > > > > > > > > > > > > > > > > > > > process > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > event > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> streams using > > Flink > > > > SQL. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> This is an > example > > > > about > > > > > > how > > > > > > > > to > > > > > > > > > > use > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > catalog: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Yijie has also > > > > written a > > > > > > > blog > > > > > > > > > post > > > > > > > > > > > > > > > explaining > > > > > > > > > > > > > > > > > why > > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > > > > > > re-implement > > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > flink > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector with > > Flink > > > > 1.9 > > > > > > and > > > > > > > > > what > > > > > > > > > > > are > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > changes > > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > > made > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> We believe > Pulsar > > is > > > > not > > > > > > > just > > > > > > > > a > > > > > > > > > > > simple > > > > > > > > > > > > > data > > > > > > > > > > > > > > > > sink > > > > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > > > > source > > > > > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > > > > > Flink. > > > > > > > > > > > > > > > > > > > > > > > > > > > > It > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> actually can be > a > > > > fully > > > > > > > > > integrated > > > > > > > > > > > > > streaming > > > > > > > > > > > > > > > > > data > > > > > > > > > > > > > > > > > > > > > storage > > > > > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > > > > > Flink in > > > > > > > > > > > > > > > > > > > > > > > > > > > > > many > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> areas (sink, > > source, > > > > > > > > > > schema/catalog > > > > > > > > > > > > and > > > > > > > > > > > > > > > > state). > > > > > > > > > > > > > > > > > > The > > > > > > > > > > > > > > > > > > > > > > > > combination > > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> and Pulsar can > > > create > > > > a > > > > > > > great > > > > > > > > > > > > streaming > > > > > > > > > > > > > > > > > warehouse > > > > > > > > > > > > > > > > > > > > > > > architecture > > > > > > > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> streaming-first, > > > > unified > > > > > > > data > > > > > > > > > > > > > processing. > > > > > > > > > > > > > > > > Since > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > > > > > talking > > > > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> contribute > Pulsar > > > > > > > integration > > > > > > > > to > > > > > > > > > > > Flink > > > > > > > > > > > > > here, > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > also > > > > > > > > > > > > > > > > > > > > > > > > > > dedicated to > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> maintain, > improve > > > and > > > > > > evolve > > > > > > > > the > > > > > > > > > > > > > integration > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > help > > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > users > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> who use both > Flink > > > and > > > > > > > Pulsar. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hope this give > > you a > > > > bit > > > > > > > more > > > > > > > > > > > > background > > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > pulsar > > > > > > > > > > > > > > > > > > > > > > > > flink > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> integration. Let > > me > > > > know > > > > > > > what > > > > > > > > > are > > > > > > > > > > > your > > > > > > > > > > > > > > > > thoughts. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Sijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3, > > 2019 > > > at > > > > > > 11:54 > > > > > > > > AM > > > > > > > > > > Yun > > > > > > > > > > > > > Tang < > > > > > > > > > > > > > > > > > > > > > > myas...@live.com> > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I can see that > > > Pulsar > > > > > > > becomes > > > > > > > > > > more > > > > > > > > > > > > and > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > popular > > > > > > > > > > > > > > > > > > > > > > > recently > > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > very > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> glad > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> to see more > > people > > > > > > willing > > > > > > > to > > > > > > > > > > > > > contribute to > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > ecosystem. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Before any > > further > > > > > > > > discussion, > > > > > > > > > > > would > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > please > > > > > > > > > > > > > > > > > > > give > > > > > > > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > > > > > > > > > explanation > > > > > > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> the > relationship > > > > > between > > > > > > > this > > > > > > > > > > > thread > > > > > > > > > > > > to > > > > > > > > > > > > > > > > current > > > > > > > > > > > > > > > > > > > > > existing > > > > > > > > > > > > > > > > > > > > > > > > JIRAs > > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > > > > > pulsar > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> source [1] and > > sink > > > > [2] > > > > > > > > > > connector? > > > > > > > > > > > > > Will the > > > > > > > > > > > > > > > > > > > > > contribution > > > > > > > > > > > > > > > > > > > > > > > > > contains > > > > > > > > > > > > > > > > > > > > > > > > > > > > part > > > > > > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> those PRs or > > > totally > > > > > > > > different > > > > > > > > > > > > > > > > implementation? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1] > > > > > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9641 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2] > > > > > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9168 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > ________________________________ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> From: Yijie > Shen > > < > > > > > > > > > > > > > > > henry.yijies...@gmail.com> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday, > > > > > September > > > > > > 3, > > > > > > > > > 2019 > > > > > > > > > > > > 13:57 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> To: > > > > > dev@flink.apache.org > > > > > > < > > > > > > > > > > > > > > > > dev@flink.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Subject: > > [DISCUSS] > > > > > > > Contribute > > > > > > > > > > > Pulsar > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > > > back > > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Dear Flink > > > Community! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I would like to > > > open > > > > > the > > > > > > > > > > discussion > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > contributing > > > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> connector [0] > > back > > > to > > > > > > > Flink. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## A brief > > > > introduction > > > > > > to > > > > > > > > > Apache > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Apache > Pulsar[1] > > > is a > > > > > > > > > > multi-tenant, > > > > > > > > > > > > > > > > > > > high-performance > > > > > > > > > > > > > > > > > > > > > > > > > distributed > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> pub-sub > messaging > > > > > system. > > > > > > > > > Pulsar > > > > > > > > > > > > > includes > > > > > > > > > > > > > > > > > > multiple > > > > > > > > > > > > > > > > > > > > > > features > > > > > > > > > > > > > > > > > > > > > > > > > such > > > > > > > > > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> native support > > for > > > > > > multiple > > > > > > > > > > > clusters > > > > > > > > > > > > > in a > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > instance, > > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> seamless > > > > > geo-replication > > > > > > of > > > > > > > > > > > messages > > > > > > > > > > > > > across > > > > > > > > > > > > > > > > > > > clusters, > > > > > > > > > > > > > > > > > > > > > > very > > > > > > > > > > > > > > > > > > > > > > > > low > > > > > > > > > > > > > > > > > > > > > > > > > > > > publish > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and end-to-end > > > > latency, > > > > > > > > > seamless > > > > > > > > > > > > > > > scalability > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > over a > > > > > > > > > > > > > > > > > > > > > > > > million > > > > > > > > > > > > > > > > > > > > > > > > > > > > topics, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and guaranteed > > > > message > > > > > > > > delivery > > > > > > > > > > > with > > > > > > > > > > > > > > > > persistent > > > > > > > > > > > > > > > > > > > message > > > > > > > > > > > > > > > > > > > > > > > > storage > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> provided by > > Apache > > > > > > > > BookKeeper. > > > > > > > > > > > > > Nowadays, > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > has > > > > > > > > > > > > > > > > > > > > > been > > > > > > > > > > > > > > > > > > > > > > > > > adopted > > > > > > > > > > > > > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> more and more > > > > > > companies[2]. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## The status > of > > > > Pulsar > > > > > > > Flink > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> The Pulsar > Flink > > > > > > connector > > > > > > > we > > > > > > > > > are > > > > > > > > > > > > > planning > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > contribute > > > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > > > built > > > > > > > > > > > > > > > > > > > > > > > > > > > > upon > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and > > > > Pulsar > > > > > > > 2.4.0. > > > > > > > > > The > > > > > > > > > > > > main > > > > > > > > > > > > > > > > features > > > > > > > > > > > > > > > > > > > are: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a > > > > streaming > > > > > > > > source > > > > > > > > > > with > > > > > > > > > > > > > > > > > exactly-once > > > > > > > > > > > > > > > > > > > > > > guarantee. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Sink > streaming > > > > > results > > > > > > to > > > > > > > > > > Pulsar > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > at-least-once > > > > > > > > > > > > > > > > > > > > > > > > > semantics. > > > > > > > > > > > > > > > > > > > > > > > > > > (We > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> would update > this > > > to > > > > > > > > > exactly-once > > > > > > > > > > > as > > > > > > > > > > > > > well > > > > > > > > > > > > > > > > when > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > gets > > > > > > > > > > > > > > > > > > > > > > > > all > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> transaction > > > features > > > > > > ready > > > > > > > in > > > > > > > > > its > > > > > > > > > > > > 2.5.0 > > > > > > > > > > > > > > > > > version) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Build upon > > Flink > > > > new > > > > > > > Table > > > > > > > > > API > > > > > > > > > > > Type > > > > > > > > > > > > > > > system > > > > > > > > > > > > > > > > > > > > > > (FLIP-37[3]), > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> automatically > > > > > > (de)serialize > > > > > > > > > > > messages > > > > > > > > > > > > > with > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > help > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > > > schema. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Integrate > with > > > > Flink > > > > > > new > > > > > > > > > > Catalog > > > > > > > > > > > > API > > > > > > > > > > > > > > > > > > > (FLIP-30[4]), > > > > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > > > > > > > > > enables > > > > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> use of Pulsar > > > topics > > > > as > > > > > > > > tables > > > > > > > > > in > > > > > > > > > > > > > Table API > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > well as > > > > > > > > > > > > > > > > > > > > > > SQL > > > > > > > > > > > > > > > > > > > > > > > > > > client. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## Reference > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [0] > > > > > > > > > > > > > > > > > > https://github.com/streamnative/pulsar-flink > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1] > > > > > > > > https://pulsar.apache.org/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2] > > > > > > > > > > > > > > > https://pulsar.apache.org/en/powered-by/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [3] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [4] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
