Thanks Becket. I think it is better for the Flink community to judge the benefits of doing this. I was trying to provide some views from outsiders.
Thanks, Sijie On Fri, Sep 20, 2019 at 10:25 AM Becket Qin <becket....@gmail.com> wrote: > Hi Sijie, > > Yes, we will have to support existing old connectors and new connectors in > parallel for a while. We have to take that maintenance overhead because > existing connectors have been used by the users for a long time. I guess It > may take at least a year for us to fully remove the old connectors. > > Process wise, we can do the same for Pulsar connector. But I am not sure if > we want to have the same burden on Pulsar connector, and I would like to > understand the benefit of doing that. > > For users, the benefit of having the old Pulsar connector checked in seems > limited because 1) that code base will be immediately deprecated in the > next release in 3-4 months; 2) users can always use it even if it is not in > the Flink code base. Admittedly it is not as convenient as having it in > Flink code base, but doesn't seem super either. And after 3-4 months, users > can just use the new connector in Flink repo. > > For Flink developers, the old connector code base is not something that we > want to evolve later. Instead, these code will be deprecated and > removed. So why do we want to get a beta version out to attract people to > use something we don't want to maintain? > > Thanks, > > Jiangjie (Becket) Qin > > > > On Fri, Sep 20, 2019 at 10:12 AM Sijie Guo <guosi...@gmail.com> wrote: > > > Thanks everyone here. Sorry for jumping into the discussion here. > > > > I am not very familiar about the deprecation process in Flink. If I > > misunderstood the process, please fix me. > > > > As far as I understand, FLIP-27 is introducing a new unified API for > > connectors. After it introduces the new API > > and before moving all the existing connectors from old API to new API, > both > > old ApI and new API will co-exist > > for a while until Flink moves all existing connectors to new API. So the > > Pulsar connector (using old API) can > > follow the deprecation process with other connector using old API and the > > deprecation of old API, no? > > > > If that's the case, I think contributing the current connector back to > > Flink rather than maintaining it outside Flink > > would provide a bit more benefits. We can deprecate the existing > > streamnative/pulsar-flink repo and point the users > > to use the connector in Flink repo. So all the review processes will > happen > > within Flink for both old connector and > > new connector. It also reduces the confusions for the users as the > > documentation and code base happen in one place. > > > > Thoughts? > > > > - Sijie > > > > > > > > > > On Fri, Sep 20, 2019 at 12:53 AM Becket Qin <becket....@gmail.com> > wrote: > > > > > Thanks for the explanation, Stephan. I have a few questions / thoughts. > > > > > > So that means we will remove the old connector without a major version > > > bump, is that correct? > > > > > > I am not 100% sure if mixing 1.10 connectors with 1.11 connectors will > > > always work because we saw some dependency class collisions in the > past. > > To > > > make it safe we may have to maintain the old code for one more release. > > > > > > To be honest I am still wondering if we have to put the old connector > in > > > Flink repo. if we check in the old connector to Flink. We will end up > in > > > the following situation: > > > 1. Old connector in streamnative/pulsar-flink repo. > > > 2. Old connector in Flink Repo, which may be different from the one in > > > Pulsar repo. (Added in 1.10, deprecated in 1.11, removed in 1.12) > > > 3. New connector in Flink Repo. > > > > > > We need to think about how to make the users in each case happy. > > > - For users of (1), I assume Sijie and Yijie will have to maintain the > > code > > > a bit longer for its own compatibility even after we have (2). In that > > > case, bugs found in old connector may or may not need to be fixed in > both > > > Flink and the streamnative/pulsar-flink repo. > > > - For users of (2), will we provide bug fixes? If we do, it will be a > > > little awkward because those bug fixes will be immediately deprecated > in > > > 1.11, and removed in 1.12. So we are essentially asking users to > migrate > > > away from the bug fix. After Flink 1.12, users may still have to switch > > to > > > use (3) due to the potential dependency class conflicts mentioned > above. > > > - Users of (3) have a much easier life and don't need to worry too > much. > > > > > > The above story seems a little complicated to tell. I think it will be > > much > > > easier to not have (2) at all. > > > 1. Old connector in streamnative/pulsar-flink repo. > > > 3. New connector in Flink Repo. > > > > > > - Old connector will only be maintained in streamnative/pulsar-flink > repo > > > until it is fully deprecated. Users can always use the existing Pulsar > > > connector in that repo. > > > - New connector will be in Flink repo and maintained like the other > > > connectors. > > > > > > This seems much simpler for users understand and they will not be > blocked > > > from using the old connector. If the concern is about the quality of > the > > > connector in streamnative/pulsar-flink repo, is it enough for us just > to > > > review the code in streamnative/pulsar-flink connector to make sure it > > > looks good from Flink's perspective? > > > > > > What do you think? > > > > > > Thanks, > > > > > > Jiangjie (Becket) Qin > > > > > > > > > On Thu, Sep 19, 2019 at 6:58 PM Stephan Ewen <se...@apache.org> wrote: > > > > > > > My take would be the following: > > > > > > > > - If we merge the connector now and replace it with a FLIP-27 > version > > > > before the 1.10 release, then we need no deprecation process > > > > - If we don't manage to replace it with a FLIP-27 version before > the > > > 1.10 > > > > release, than it is good that we have the other version, so no users > > get > > > > blocked. > > > > > > > > In the latter case we can see how we want to do it. Immediate removal > > of > > > > the old version or deprecation label and keeping it for one more > > release. > > > > Given that you should be able to use a Flink 1.10 connector with > Flink > > > 1.11 > > > > as well (stable public APIs) there is also a workaround if you need > an > > > old > > > > connector in a newer version. So immediate removal might even be > > > feasible. > > > > > > > > > > > > On Thu, Sep 19, 2019 at 11:09 AM Becket Qin <becket....@gmail.com> > > > wrote: > > > > > > > > > Hi Stephan, > > > > > > > > > > Thanks for the clarification. I completely agree with you and > Thomas > > on > > > > the > > > > > process of adding connectors to Flink repo. However, I am wondering > > > what > > > > is > > > > > the deprecation process? Given the main concern here was that we > may > > > have > > > > > to maintain two Pulsar connector code bases until the old one is > > > removed > > > > > from the repo, it would be good to know how long we have to do > that. > > > > > > > > > > Thanks, > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen <se...@apache.org> > > wrote: > > > > > > > > > > > Some quick thoughts on the connector contribution process. I > > > basically > > > > > > reiterate here what Thomas mentioned in another thread about the > > > > Kinesis > > > > > > connector. > > > > > > > > > > > > For connectors, we should favor a low-overhead contribution > > process, > > > > and > > > > > > accept user code and changes more readily than in the core > system. > > > > > > That is because connectors have both a big variety of scenarios > > they > > > > get > > > > > > used in (only through use and many small contributions do they > > become > > > > > > really useful over time) and at the same time, and committers do > > not > > > > use > > > > > > the connector themselves and usually cannot foresee too well what > > is > > > > > > needed. > > > > > > > > > > > > Further more, a missing connector (or connector feature) is > often a > > > > > bigger > > > > > > show stopper for users than a missing API or system feature. > > > > > > > > > > > > Along these lines of thougt, the conclusion would be to take the > > > Pulsar > > > > > > connector now, focus the review on legal/dependencies/rough code > > > style > > > > > and > > > > > > conventions, label it as "beta" (in the sense of "new code" that > is > > > > "not > > > > > > yet tested through longer use") and go ahead. And then evolve it > > > > quickly > > > > > > without putting formal blockers in the way, meaning also adding a > > new > > > > > FLIP > > > > > > 27 version when it is there. > > > > > > > > > > > > Best, > > > > > > Stephan > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <becket....@gmail.com > > > > > > wrote: > > > > > > > > > > > > > Hi Yijie, > > > > > > > > > > > > > > Could you please follow the FLIP process to start a new FLIP > > > > > [DISCUSSION] > > > > > > > thread in the mailing list? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process > > > > > > > > > > > > > > I see two FLIP-69 discussion in the mailing list now. So there > > is a > > > > > FLIP > > > > > > > number collision. Can you change the FLIP number to 72? > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong < > walter...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > Hi Yijie, > > > > > > > > > > > > > > > > Thanks for sharing the pulsar FLIP. > > > > > > > > Would you mind enabling comments/suggestions on the google > doc > > > > link? > > > > > > This > > > > > > > > way the contributors from the community can comment on the > doc. > > > > > > > > > > > > > > > > Best, > > > > > > > > Rong > > > > > > > > > > > > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen < > > > > > henry.yijies...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > > > > > > > I've drafted a FLIP that describes the current design of > the > > > > Pulsar > > > > > > > > > connector: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit# > > > > > > > > > > > > > > > > > > Please take a look and let me know what you think. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Yijie > > > > > > > > > > > > > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong < > > > walter...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > Sorry for joining the discussion late and thanks Yijie & > > > Sijie > > > > > for > > > > > > > > > driving > > > > > > > > > > the discussion. > > > > > > > > > > I also think the Pulsar connector would be a very > valuable > > > > > addition > > > > > > > to > > > > > > > > > > Flink. I can also help out a bit on the review side :-) > > > > > > > > > > > > > > > > > > > > Regarding the timeline, I also share concerns with Becket > > on > > > > the > > > > > > > > > > relationship between the new Pulsar connector and > FLIP-27. > > > > > > > > > > There's also another discussion just started by Stephan > on > > > > > dropping > > > > > > > > Kafka > > > > > > > > > > 9/10 support on next Flink release [1]. Although the > > > situation > > > > > is > > > > > > > > > somewhat > > > > > > > > > > different, and Kafka 9/10 connector has been in Flink for > > > > almost > > > > > > 3-4 > > > > > > > > > years, > > > > > > > > > > based on the discussion I am not sure if a major version > > > > release > > > > > > is a > > > > > > > > > > requirement for removing old connector supports. > > > > > > > > > > > > > > > > > > > > I think there shouldn't be a blocker if we agree the old > > > > > connector > > > > > > > will > > > > > > > > > be > > > > > > > > > > removed once FLIP-27 based Pulsar connector is there. As > > > > Stephan > > > > > > > > stated, > > > > > > > > > it > > > > > > > > > > is easier to contribute the source sooner and adjust it > > > later. > > > > > > > > > > We should also ensure we clearly communicate the message: > > for > > > > > > > example, > > > > > > > > > > putting an experimental flag on the pre-FLIP27 connector > > page > > > > of > > > > > > the > > > > > > > > > > website, documentations, etc. Any other thoughts? > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Rong > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin < > > > > becket....@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Technically speaking, removing the old connector code > is > > a > > > > > > > backwards > > > > > > > > > > > incompatible change which requires a major version > bump, > > > i.e. > > > > > > Flink > > > > > > > > > 2.x. > > > > > > > > > > > Given that we don't have a clear plan on when to have > the > > > > next > > > > > > > major > > > > > > > > > > > version release, it seems unclear how long the old > > > connector > > > > > code > > > > > > > > will > > > > > > > > > be > > > > > > > > > > > there if we check it in right now. Or will we remove > the > > > old > > > > > > > > connector > > > > > > > > > > > without a major version bump? In any case, it sounds > not > > > > quite > > > > > > user > > > > > > > > > > > friendly to the those who might use the old Pulsar > > > > connector. I > > > > > > am > > > > > > > > not > > > > > > > > > sure > > > > > > > > > > > if it is worth these potential problems in order to > have > > > the > > > > > > Pulsar > > > > > > > > > source > > > > > > > > > > > connector checked in one or two months earlier. > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen < > > > > se...@apache.org > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Agreed, if we check in the old code, we should make > it > > > > clear > > > > > > that > > > > > > > > it > > > > > > > > > will > > > > > > > > > > > > be removed as soon as the FLIP-27 based version of > the > > > > > > connector > > > > > > > is > > > > > > > > > > > there. > > > > > > > > > > > > We should not commit to maintaining the old version, > > that > > > > > would > > > > > > > be > > > > > > > > > indeed > > > > > > > > > > > > too much overhead. > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin < > > > > > > becket....@gmail.com > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi Stephan, > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the volunteering to help. > > > > > > > > > > > > > > > > > > > > > > > > > > Yes, the overhead would just be review capacity. In > > > > fact, I > > > > > > am > > > > > > > > not > > > > > > > > > > > > worrying > > > > > > > > > > > > > too much about the review capacity. That is just a > > one > > > > time > > > > > > > cost. > > > > > > > > > My > > > > > > > > > > > > > concern is mainly about the long term burden. > Assume > > we > > > > > have > > > > > > > new > > > > > > > > > source > > > > > > > > > > > > > interface ready in 1.10 with newly added Pulsar > > > > connectors > > > > > in > > > > > > > old > > > > > > > > > > > > > interface. Later on if we migrate Pulsar to new > > source > > > > > > > interface, > > > > > > > > > the > > > > > > > > > > > old > > > > > > > > > > > > > Pulsar connector might be deprecated almost > > immediately > > > > > after > > > > > > > > > checked > > > > > > > > > > > in, > > > > > > > > > > > > > but we may still have to maintain two code bases. > For > > > the > > > > > > > > existing > > > > > > > > > > > > > connectors, we have to do that anyways. But it > would > > be > > > > > good > > > > > > to > > > > > > > > > avoid > > > > > > > > > > > > > introducing a new connector with the same problem. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen < > > > > > > se...@apache.org > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nice to see this lively discussion about the > Pulsar > > > > > > > connector. > > > > > > > > > > > > > > Some thoughts on the open questions: > > > > > > > > > > > > > > > > > > > > > > > > > > > > ## Contribute to Flink or maintain as a community > > > > package > > > > > > > > > > > > > > > > > > > > > > > > > > > > Looks like the discussion is more going towards > > > > > > > contribution. I > > > > > > > > > think > > > > > > > > > > > > > that > > > > > > > > > > > > > > is good, especially if we think that we want to > > > build a > > > > > > > > similarly > > > > > > > > > > > deep > > > > > > > > > > > > > > integration with Pulsar as we have for example > with > > > > > Kafka. > > > > > > > The > > > > > > > > > > > > connector > > > > > > > > > > > > > > already looks like a more thorough connector than > > > many > > > > > > others > > > > > > > > we > > > > > > > > > have > > > > > > > > > > > > in > > > > > > > > > > > > > > the repository. > > > > > > > > > > > > > > > > > > > > > > > > > > > > With either a repo split, or the new build > system, > > I > > > > hope > > > > > > > that > > > > > > > > > the > > > > > > > > > > > > build > > > > > > > > > > > > > > overhead is not a problem. > > > > > > > > > > > > > > > > > > > > > > > > > > > > ## Committer Support > > > > > > > > > > > > > > > > > > > > > > > > > > > > Becket offered some help already, I can also > help a > > > > bit. > > > > > I > > > > > > > hope > > > > > > > > > that > > > > > > > > > > > > > > between us, we can cover this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > ## Contribute now, or wait for FLIP-27 > > > > > > > > > > > > > > > > > > > > > > > > > > > > As Becket said, FLIP-27 is actually making some > > > PoC-ing > > > > > > > > > progress, but > > > > > > > > > > > > > will > > > > > > > > > > > > > > take 2 more months, I would estimate, before it > is > > > > fully > > > > > > > > > available. > > > > > > > > > > > > > > > > > > > > > > > > > > > > If we want to be on the safe side with the > > > > contribution, > > > > > we > > > > > > > > > should > > > > > > > > > > > > > > contribute the source sooner and adjust it later. > > > That > > > > > > would > > > > > > > > also > > > > > > > > > > > help > > > > > > > > > > > > us > > > > > > > > > > > > > > in case things get crazy towards the 1.10 feature > > > > freeze > > > > > > and > > > > > > > it > > > > > > > > > would > > > > > > > > > > > > be > > > > > > > > > > > > > > hard to find time to review the new changes. > > > > > > > > > > > > > > What would be the overhead of contributing now? > > Given > > > > > that > > > > > > > the > > > > > > > > > code > > > > > > > > > > > is > > > > > > > > > > > > > > already there, it looks like it would be only > > review > > > > > > > capacity, > > > > > > > > > right? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > Stephan > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen < > > > > > > > > > > > henry.yijies...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your attention and the promotion of > > this > > > > > work. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We will prepare a FLIP as soon as possible for > > more > > > > > > > specific > > > > > > > > > > > > > discussions. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For FLIP-27, it seems that we have not reached > a > > > > > > consensus. > > > > > > > > > > > > Therefore, > > > > > > > > > > > > > > > I will explain all the functionalities of the > > > > existing > > > > > > > > > connector in > > > > > > > > > > > > > > > the FLIP (including Source, Sink, and Catalog) > to > > > > > > continue > > > > > > > > our > > > > > > > > > > > > > > > discussions in FLIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your kind help. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > Yijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin < > > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Sijie, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If we agree that the goal is to have Pulsar > > > > connector > > > > > > in > > > > > > > > > 1.10, > > > > > > > > > > > how > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > do the following: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar connector to > > Flink > > > > main > > > > > > > repo > > > > > > > > > as it > > > > > > > > > > > > is a > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > public interface to Flink main repo. > > > > > > > > > > > > > > > > 1. Start to review the Pulsar sink right away > > as > > > > > there > > > > > > is > > > > > > > > no > > > > > > > > > > > change > > > > > > > > > > > > > to > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > sink interface so far. > > > > > > > > > > > > > > > > 2. Wait a little bit on FLIP-27. Flink 1.10 > is > > > > going > > > > > to > > > > > > > be > > > > > > > > > code > > > > > > > > > > > > > freeze > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > late Nov and let's say we give a month to the > > > > > > development > > > > > > > > and > > > > > > > > > > > > review > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > Pulsar connector, we need to have FLIP-27 by > > late > > > > > Oct. > > > > > > > > There > > > > > > > > > are > > > > > > > > > > > > > still > > > > > > > > > > > > > > 7 > > > > > > > > > > > > > > > > weeks. Personally I think it is doable. If > > > FLIP-27 > > > > is > > > > > > not > > > > > > > > > ready > > > > > > > > > > > by > > > > > > > > > > > > > late > > > > > > > > > > > > > > > > Oct, we can review and check in Pulsar > > connector > > > > with > > > > > > the > > > > > > > > > > > existing > > > > > > > > > > > > > > source > > > > > > > > > > > > > > > > interface. This means we will have Pulsar > > > connector > > > > > in > > > > > > > > Flink > > > > > > > > > > > 1.10, > > > > > > > > > > > > > > either > > > > > > > > > > > > > > > > with or without FLIP-27. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Because we are going to have Pulsar sink and > > > source > > > > > > > checked > > > > > > > > > in > > > > > > > > > > > > > > > separately, > > > > > > > > > > > > > > > > it might make sense to have two FLIPs, one > for > > > > Pulsar > > > > > > > sink > > > > > > > > > and > > > > > > > > > > > > > another > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > Pulsar source. And we can start the work on > > > Pulsar > > > > > sink > > > > > > > > right > > > > > > > > > > > away. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo < > > > > > > > > guosi...@gmail.com > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you Bowen and Becket. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What's the take from Flink community? Shall > > we > > > > wait > > > > > > for > > > > > > > > > FLIP-27 > > > > > > > > > > > > or > > > > > > > > > > > > > > > shall we > > > > > > > > > > > > > > > > > proceed to next steps? And what the next > > steps > > > > are? > > > > > > :-) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > Sijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li < > > > > > > > > > bowenl...@gmail.com> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think having a Pulsar connector in > Flink > > > can > > > > > be a > > > > > > > > good > > > > > > > > > > > mutual > > > > > > > > > > > > > > > benefit > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > both communities. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Another perspective is that Pulsar > > connector > > > is > > > > > the > > > > > > > 1st > > > > > > > > > > > > streaming > > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > that integrates with Flink's metadata > > > > management > > > > > > > system > > > > > > > > > and > > > > > > > > > > > > > Catalog > > > > > > > > > > > > > > > APIs. > > > > > > > > > > > > > > > > > > It'll be cool to see how the integration > > > turns > > > > > out > > > > > > > and > > > > > > > > > > > whether > > > > > > > > > > > > we > > > > > > > > > > > > > > > need to > > > > > > > > > > > > > > > > > > improve Flink Catalog stack, which are > > > > currently > > > > > in > > > > > > > > > Beta, to > > > > > > > > > > > > > cater > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > streaming source/sink. Thus I'm in favor > of > > > > > merging > > > > > > > > > Pulsar > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > > Flink 1.10. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'd suggest to submit smaller sized PRs, > > e.g. > > > > > maybe > > > > > > > one > > > > > > > > > for > > > > > > > > > > > > basic > > > > > > > > > > > > > > > > > > source/sink functionalities and another > for > > > > > schema > > > > > > > and > > > > > > > > > > > catalog > > > > > > > > > > > > > > > > > integration, > > > > > > > > > > > > > > > > > > just to make them easier to review. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It doesn't seem to hurt to wait for > > FLIP-27. > > > > But > > > > > I > > > > > > > > don't > > > > > > > > > > > think > > > > > > > > > > > > > > > FLIP-27 > > > > > > > > > > > > > > > > > > should be a blocker in cases where it > > cannot > > > > make > > > > > > its > > > > > > > > way > > > > > > > > > > > into > > > > > > > > > > > > > 1.10 > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > doesn't leave reasonable amount of time > for > > > > > > > committers > > > > > > > > to > > > > > > > > > > > > review > > > > > > > > > > > > > or > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > Pulsar connector to fully adapt to new > > > > > interfaces. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Bowen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket > Qin < > > > > > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > You are right. It all depends on when > the > > > new > > > > > > > source > > > > > > > > > > > > interface > > > > > > > > > > > > > is > > > > > > > > > > > > > > > going > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > be ready. Personally I think it would > be > > > > there > > > > > in > > > > > > > > > about a > > > > > > > > > > > > month > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > so. > > > > > > > > > > > > > > > > > > But > > > > > > > > > > > > > > > > > > > I could be too optimistic. It would > also > > be > > > > > good > > > > > > to > > > > > > > > > hear > > > > > > > > > > > what > > > > > > > > > > > > > do > > > > > > > > > > > > > > > > > Aljoscha > > > > > > > > > > > > > > > > > > > and Stephan think as they are also > > involved > > > > in > > > > > > > > FLIP-27. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In general I think we should have > Pulsar > > > > > > connector > > > > > > > in > > > > > > > > > Flink > > > > > > > > > > > > > 1.10, > > > > > > > > > > > > > > > > > > > preferably with the new source > interface. > > > We > > > > > can > > > > > > > also > > > > > > > > > check > > > > > > > > > > > > it > > > > > > > > > > > > > in > > > > > > > > > > > > > > > right > > > > > > > > > > > > > > > > > > now > > > > > > > > > > > > > > > > > > > with old source interface, but I > suspect > > > few > > > > > > users > > > > > > > > > will use > > > > > > > > > > > > it > > > > > > > > > > > > > > > before > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > next official release. Therefore, it > > seems > > > > > > > reasonable > > > > > > > > > to > > > > > > > > > > > > wait a > > > > > > > > > > > > > > > little > > > > > > > > > > > > > > > > > > bit > > > > > > > > > > > > > > > > > > > to see whether we can jump to the new > > > source > > > > > > > > > interface. As > > > > > > > > > > > > long > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > make > > > > > > > > > > > > > > > > > > > sure Flink 1.10 has it, waiting a > little > > > bit > > > > > > > doesn't > > > > > > > > > seem > > > > > > > > > > > to > > > > > > > > > > > > > hurt > > > > > > > > > > > > > > > much. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till > > > Rohrmann > > > > < > > > > > > > > > > > > > > trohrm...@apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm wondering what the problem would > be > > > if > > > > we > > > > > > > > > committed > > > > > > > > > > > the > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > connector before the new source > > interface > > > > is > > > > > > > ready. > > > > > > > > > If I > > > > > > > > > > > > > > > understood > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > correctly, then we need to support > the > > > old > > > > > > source > > > > > > > > > > > interface > > > > > > > > > > > > > > > anyway > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > existing connectors. By checking it > in > > > > early > > > > > I > > > > > > > > could > > > > > > > > > see > > > > > > > > > > > > the > > > > > > > > > > > > > > > benefit > > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > > our users could start using the > > connector > > > > > > > earlier. > > > > > > > > > > > > Moreover, > > > > > > > > > > > > > it > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > > > > prevent that the Pulsar integration > is > > > > being > > > > > > > > delayed > > > > > > > > > in > > > > > > > > > > > > case > > > > > > > > > > > > > > > that the > > > > > > > > > > > > > > > > > > > > source interface should be delayed. > The > > > > only > > > > > > > > > downside I > > > > > > > > > > > see > > > > > > > > > > > > > is > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > extra > > > > > > > > > > > > > > > > > > > > review effort and potential fixes > which > > > > might > > > > > > be > > > > > > > > > > > irrelevant > > > > > > > > > > > > > for > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > > source interface implementation. I > > guess > > > it > > > > > > > mainly > > > > > > > > > > > depends > > > > > > > > > > > > on > > > > > > > > > > > > > > how > > > > > > > > > > > > > > > > > > certain > > > > > > > > > > > > > > > > > > > > we are when the new source interface > > will > > > > be > > > > > > > ready. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM Becket > > > Qin < > > > > > > > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Sijie and Yijie, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for sharing your thoughts. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Just want to have some update on > > > FLIP-27. > > > > > > > > Although > > > > > > > > > the > > > > > > > > > > > > FLIP > > > > > > > > > > > > > > > wiki > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > discussion thread has been quiet > for > > > some > > > > > > > time, a > > > > > > > > > few > > > > > > > > > > > > > > > committer / > > > > > > > > > > > > > > > > > > > > > contributors in Flink community > were > > > > > actually > > > > > > > > > > > prototyping > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > entire > > > > > > > > > > > > > > > > > > > > thing. > > > > > > > > > > > > > > > > > > > > > We have made some good progress > there > > > but > > > > > > want > > > > > > > to > > > > > > > > > > > update > > > > > > > > > > > > > the > > > > > > > > > > > > > > > FLIP > > > > > > > > > > > > > > > > > > wiki > > > > > > > > > > > > > > > > > > > > > after the entire thing is verified > to > > > > work > > > > > in > > > > > > > > case > > > > > > > > > > > there > > > > > > > > > > > > > are > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > last > > > > > > > > > > > > > > > > > > > > > minute surprise in the > > implementation. > > > I > > > > > > don't > > > > > > > > > have an > > > > > > > > > > > > > exact > > > > > > > > > > > > > > > ETA > > > > > > > > > > > > > > > > > yet, > > > > > > > > > > > > > > > > > > > > but I > > > > > > > > > > > > > > > > > > > > > guess it is going to be within a > > month > > > or > > > > > so. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am happy to review the current > > Flink > > > > > Pulsar > > > > > > > > > connector > > > > > > > > > > > > and > > > > > > > > > > > > > > > see if > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > > > > > fit in FLIP-27. It would be good to > > > avoid > > > > > the > > > > > > > > case > > > > > > > > > that > > > > > > > > > > > > we > > > > > > > > > > > > > > > checked > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > Pulsar connector with some review > > > efforts > > > > > and > > > > > > > > > shortly > > > > > > > > > > > > after > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > > > Source interface is ready. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM > Yijie > > > > Shen < > > > > > > > > > > > > > > > > > henry.yijies...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for all the feedback and > > > > > > suggestions! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As Sijie said, the goal of the > > > > connector > > > > > > has > > > > > > > > > always > > > > > > > > > > > > been > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > provide > > > > > > > > > > > > > > > > > > > > > > users with the latest features of > > > both > > > > > > > systems > > > > > > > > as > > > > > > > > > > > soon > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > possible. > > > > > > > > > > > > > > > > > > > We > > > > > > > > > > > > > > > > > > > > > > propose to contribute the > connector > > > to > > > > > > Flink > > > > > > > > and > > > > > > > > > hope > > > > > > > > > > > > to > > > > > > > > > > > > > > get > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > > > suggestions and feedback from > Flink > > > > > experts > > > > > > > to > > > > > > > > > ensure > > > > > > > > > > > > the > > > > > > > > > > > > > > > high > > > > > > > > > > > > > > > > > > > quality > > > > > > > > > > > > > > > > > > > > > > of the connector. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For FLIP-27, we noticed its > > existence > > > > at > > > > > > the > > > > > > > > > > > beginning > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > reworking > > > > > > > > > > > > > > > > > > > > > > the connector implementation > based > > on > > > > > Flink > > > > > > > > 1.9; > > > > > > > > > we > > > > > > > > > > > > also > > > > > > > > > > > > > > > wanted > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > build a connector that supports > > both > > > > > batch > > > > > > > and > > > > > > > > > stream > > > > > > > > > > > > > > > computing > > > > > > > > > > > > > > > > > > based > > > > > > > > > > > > > > > > > > > > > > on it. > > > > > > > > > > > > > > > > > > > > > > However, it has been inactive for > > > some > > > > > > time, > > > > > > > so > > > > > > > > > we > > > > > > > > > > > > > decided > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > provide > > > > > > > > > > > > > > > > > > > > > > a connector with most of the new > > > > > features, > > > > > > > such > > > > > > > > > as > > > > > > > > > > > the > > > > > > > > > > > > > new > > > > > > > > > > > > > > > type > > > > > > > > > > > > > > > > > > > system > > > > > > > > > > > > > > > > > > > > > > and the new catalog API first. We > > > will > > > > > pay > > > > > > > > > attention > > > > > > > > > > > to > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > progress > > > > > > > > > > > > > > > > > > > > > > of FLIP-27 continually and > > > incorporate > > > > it > > > > > > > with > > > > > > > > > the > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > soon > > > > > > > > > > > > > > > > > > > > > > as possible. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regarding the test status of the > > > > > connector, > > > > > > > we > > > > > > > > > are > > > > > > > > > > > > > > following > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > other > > > > > > > > > > > > > > > > > > > > > > connectors' test in Flink > > repository > > > > and > > > > > > > aimed > > > > > > > > to > > > > > > > > > > > > provide > > > > > > > > > > > > > > > > > > throughout > > > > > > > > > > > > > > > > > > > > > > tests as we could. We are also > > happy > > > to > > > > > > hear > > > > > > > > > > > > suggestions > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > supervision from the Flink > > community > > > to > > > > > > > improve > > > > > > > > > the > > > > > > > > > > > > > > > stability and > > > > > > > > > > > > > > > > > > > > > > performance of the connector > > > > > continuously. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > > Yijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59 AM > > Sijie > > > > Guo > > > > > < > > > > > > > > > > > > > > guosi...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks everyone for the > comments > > > and > > > > > > > > feedback. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It seems to me that the main > > > question > > > > > > here > > > > > > > is > > > > > > > > > > > about - > > > > > > > > > > > > > > "how > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > community maintain the > > connector?". > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Here are two thoughts from > > myself. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1) I think how and where to > host > > > this > > > > > > > > > integration > > > > > > > > > > > is > > > > > > > > > > > > > kind > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > less > > > > > > > > > > > > > > > > > > > > > > important > > > > > > > > > > > > > > > > > > > > > > > here. I believe there can be > many > > > > ways > > > > > to > > > > > > > > > achieve > > > > > > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > > As part of the contribution, > what > > > we > > > > > are > > > > > > > > > looking > > > > > > > > > > > for > > > > > > > > > > > > > here > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > how > > > > > > > > > > > > > > > > > > > > these > > > > > > > > > > > > > > > > > > > > > > two > > > > > > > > > > > > > > > > > > > > > > > communities can build the > > > > collaboration > > > > > > > > > > > relationship > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > developing > > > > > > > > > > > > > > > > > > > > > > > the integration between Pulsar > > and > > > > > Flink. > > > > > > > > Even > > > > > > > > > we > > > > > > > > > > > can > > > > > > > > > > > > > try > > > > > > > > > > > > > > > our > > > > > > > > > > > > > > > > > > best > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > catch > > > > > > > > > > > > > > > > > > > > > > > up all the updates in Flink > > > > community. > > > > > We > > > > > > > are > > > > > > > > > still > > > > > > > > > > > > > > > > > > > > > > > facing the fact that we have > less > > > > > > > experiences > > > > > > > > > in > > > > > > > > > > > > Flink > > > > > > > > > > > > > > than > > > > > > > > > > > > > > > > > folks > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > community. In order to make > sure > > we > > > > > > > maintain > > > > > > > > > and > > > > > > > > > > > > > deliver > > > > > > > > > > > > > > > > > > > > > > > a high-quality pulsar-flink > > > > integration > > > > > > to > > > > > > > > the > > > > > > > > > > > users > > > > > > > > > > > > > who > > > > > > > > > > > > > > > use > > > > > > > > > > > > > > > > > both > > > > > > > > > > > > > > > > > > > > > > > technologies, we need some help > > > from > > > > > the > > > > > > > > > experts > > > > > > > > > > > from > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > community. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2) We have been following > FLIP-27 > > > > for a > > > > > > > > while. > > > > > > > > > > > > > Originally > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > were > > > > > > > > > > > > > > > > > > > > > > thinking > > > > > > > > > > > > > > > > > > > > > > > of contributing the connectors > > back > > > > > after > > > > > > > > > > > integrating > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > new API introduced in FLIP-27. > > But > > > we > > > > > > > decided > > > > > > > > > to > > > > > > > > > > > > > initiate > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > conversation > > > > > > > > > > > > > > > > > > > > > > > as early as possible. Because > we > > > > > believe > > > > > > > > there > > > > > > > > > are > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > benefits > > > > > > > > > > > > > > > > > > > > doing > > > > > > > > > > > > > > > > > > > > > > > it now rather than later. As > part > > > of > > > > > > > > > contribution, > > > > > > > > > > > it > > > > > > > > > > > > > can > > > > > > > > > > > > > > > help > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > community understand more about > > > > Pulsar > > > > > > and > > > > > > > > the > > > > > > > > > > > > > potential > > > > > > > > > > > > > > > > > > > integration > > > > > > > > > > > > > > > > > > > > > > points. > > > > > > > > > > > > > > > > > > > > > > > Also we can also help Flink > > > community > > > > > > > verify > > > > > > > > > the > > > > > > > > > > > new > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > API > > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > > well > > > > > > > > > > > > > > > > > > > > > > > as other new API (e.g. catalog > > > API). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > Sijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24 AM > > > Becket > > > > > > Qin < > > > > > > > > > > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yijie, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the interest in > > > > > contributing > > > > > > > the > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > connector. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In general, I think having > > Pulsar > > > > > > > connector > > > > > > > > > with > > > > > > > > > > > > > strong > > > > > > > > > > > > > > > > > support > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > > > > valuable addition to Flink. > So > > I > > > am > > > > > > happy > > > > > > > > the > > > > > > > > > > > > > shepherd > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > > effort. > > > > > > > > > > > > > > > > > > > > > > > > Meanwhile, I would also like > to > > > > > provide > > > > > > > > some > > > > > > > > > > > > context > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > recent > > > > > > > > > > > > > > > > > > > > > > efforts on > > > > > > > > > > > > > > > > > > > > > > > > the Flink connectors > ecosystem. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The current way Flink > maintains > > > its > > > > > > > > > connector has > > > > > > > > > > > > hit > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > scalability > > > > > > > > > > > > > > > > > > > > > > bar. > > > > > > > > > > > > > > > > > > > > > > > > With more and more connectors > > > > coming > > > > > > into > > > > > > > > > Flink > > > > > > > > > > > > repo, > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > > facing a > > > > > > > > > > > > > > > > > > > > > > few > > > > > > > > > > > > > > > > > > > > > > > > problems such as long build > and > > > > > testing > > > > > > > > > time. To > > > > > > > > > > > > > > address > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > > > > problem, > > > > > > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > > > > > have attempted to do the > > > following: > > > > > > > > > > > > > > > > > > > > > > > > 1. Split out the connectors > > into > > > a > > > > > > > separate > > > > > > > > > > > > > repository. > > > > > > > > > > > > > > > This > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > temporarily > > > > > > > > > > > > > > > > > > > > > > > > on hold due to potential > > solution > > > > to > > > > > > > > shorten > > > > > > > > > the > > > > > > > > > > > > > build > > > > > > > > > > > > > > > time. > > > > > > > > > > > > > > > > > > > > > > > > 2. Encourage the connectors > to > > > stay > > > > > as > > > > > > > > > ecosystem > > > > > > > > > > > > > > project > > > > > > > > > > > > > > > > > while > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > tries > > > > > > > > > > > > > > > > > > > > > > > > to provide good support for > > > > > > functionality > > > > > > > > and > > > > > > > > > > > > > > > compatibility > > > > > > > > > > > > > > > > > > > tests. > > > > > > > > > > > > > > > > > > > > > > Robert > > > > > > > > > > > > > > > > > > > > > > > > has driven to create a Flink > > > > > Ecosystem > > > > > > > > > project > > > > > > > > > > > > > website > > > > > > > > > > > > > > > and it > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > going > > > > > > > > > > > > > > > > > > > > > > > > through some final approval > > > > process. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Given the above efforts, it > > would > > > > be > > > > > > > great > > > > > > > > to > > > > > > > > > > > first > > > > > > > > > > > > > see > > > > > > > > > > > > > > > if we > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector as an > > ecosystem > > > > > > project > > > > > > > > with > > > > > > > > > > > great > > > > > > > > > > > > > > > support. > > > > > > > > > > > > > > > > > It > > > > > > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > > > good to hear how the Flink > > Pulsar > > > > > > > connector > > > > > > > > > is > > > > > > > > > > > > tested > > > > > > > > > > > > > > > > > currently > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > see > > > > > > > > > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > > > > > > > > > we can learn something to > > > maintain > > > > it > > > > > > as > > > > > > > an > > > > > > > > > > > > ecosystem > > > > > > > > > > > > > > > project > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > good > > > > > > > > > > > > > > > > > > > > > > > > quality and test coverage. If > > the > > > > > > quality > > > > > > > > as > > > > > > > > > an > > > > > > > > > > > > > > ecosystem > > > > > > > > > > > > > > > > > > project > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > hard > > > > > > > > > > > > > > > > > > > > > > > > to guarantee, we may as well > > > adopt > > > > it > > > > > > > into > > > > > > > > > the > > > > > > > > > > > main > > > > > > > > > > > > > > repo. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > BTW, another ongoing effort > is > > > > > FLIP-27 > > > > > > > > where > > > > > > > > > we > > > > > > > > > > > are > > > > > > > > > > > > > > > making > > > > > > > > > > > > > > > > > > > changes > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > Flink source connector > > > architecture > > > > > and > > > > > > > > > > > interface. > > > > > > > > > > > > > This > > > > > > > > > > > > > > > > > change > > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > > > > likely > > > > > > > > > > > > > > > > > > > > > > > > land in 1.10. Therefore > timing > > > > wise, > > > > > if > > > > > > > we > > > > > > > > > are > > > > > > > > > > > > going > > > > > > > > > > > > > to > > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > connector in main repo, I am > > > > > wondering > > > > > > if > > > > > > > > we > > > > > > > > > > > should > > > > > > > > > > > > > > hold > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > little > > > > > > > > > > > > > > > > > > > > bit > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > let the Pulsar connector > adapt > > to > > > > the > > > > > > new > > > > > > > > > > > interface > > > > > > > > > > > > > to > > > > > > > > > > > > > > > avoid > > > > > > > > > > > > > > > > > > > > shortly > > > > > > > > > > > > > > > > > > > > > > > > deprecated work? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 4:32 > PM > > > > > Chesnay > > > > > > > > > Schepler < > > > > > > > > > > > > > > > > > > > > ches...@apache.org> > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm quite worried that we > may > > > end > > > > > up > > > > > > > > > repeating > > > > > > > > > > > > > > history. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There were already 2 > attempts > > > at > > > > > > > > > contributing a > > > > > > > > > > > > > > pulsar > > > > > > > > > > > > > > > > > > > connector, > > > > > > > > > > > > > > > > > > > > > > both > > > > > > > > > > > > > > > > > > > > > > > > > of which failed because no > > > > > committer > > > > > > > was > > > > > > > > > > > getting > > > > > > > > > > > > > > > involved, > > > > > > > > > > > > > > > > > > > > despite > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > contributor opening a > > dedicated > > > > > > > > discussion > > > > > > > > > > > thread > > > > > > > > > > > > > > > about the > > > > > > > > > > > > > > > > > > > > > > contribution > > > > > > > > > > > > > > > > > > > > > > > > > beforehand and getting > > several > > > > +1's > > > > > > > from > > > > > > > > > > > > > committers. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We should really make sure > > that > > > > if > > > > > we > > > > > > > > > > > > > welcome/approve > > > > > > > > > > > > > > > such > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > > > > > contribution it will > actually > > > get > > > > > the > > > > > > > > > attention > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > deserves. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As such, I'm inclined to > > > > recommend > > > > > > > > > maintaining > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > > > outside > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > Flink. We could link to it > > from > > > > the > > > > > > > > > > > documentation > > > > > > > > > > > > > to > > > > > > > > > > > > > > > give > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > > > > > exposure. > > > > > > > > > > > > > > > > > > > > > > > > > With the upcoming page for > > > > sharing > > > > > > > > > artifacts > > > > > > > > > > > > among > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > community > > > > > > > > > > > > > > > > > > > > > > (what's > > > > > > > > > > > > > > > > > > > > > > > > > the state of that anyway?), > > > this > > > > > may > > > > > > > be a > > > > > > > > > > > better > > > > > > > > > > > > > > > option. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 04/09/2019 10:16, Till > > > > Rohrmann > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks a lot for starting > > > this > > > > > > > > discussion > > > > > > > > > > > > Yijie. > > > > > > > > > > > > > I > > > > > > > > > > > > > > > think > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > > > connector would be a very > > > > > valuable > > > > > > > > > addition > > > > > > > > > > > > since > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > becomes > > > > > > > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > more popular and it would > > > > further > > > > > > > > expand > > > > > > > > > > > > Flink's > > > > > > > > > > > > > > > > > > > > > interoperability. > > > > > > > > > > > > > > > > > > > > > > Also > > > > > > > > > > > > > > > > > > > > > > > > > > from a project > perspective > > it > > > > > makes > > > > > > > > > sense for > > > > > > > > > > > > me > > > > > > > > > > > > > to > > > > > > > > > > > > > > > place > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > > the downstream project. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My main concern/question > is > > > how > > > > > can > > > > > > > the > > > > > > > > > Flink > > > > > > > > > > > > > > > community > > > > > > > > > > > > > > > > > > > > maintain > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > connector? We have seen > in > > > the > > > > > past > > > > > > > > that > > > > > > > > > > > > > connectors > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > most > > > > > > > > > > > > > > > > > > > > > > > > > > actively developed > > components > > > > > > because > > > > > > > > > they > > > > > > > > > > > need > > > > > > > > > > > > > to > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > kept > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > sync > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > external system and with > > > Flink. > > > > > > Given > > > > > > > > > that > > > > > > > > > > > the > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > community > > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > > > willing > > > > > > > > > > > > > > > > > > > > > > > > > > to help with maintaining, > > > > > improving > > > > > > > and > > > > > > > > > > > > evolving > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > connector, > > > > > > > > > > > > > > > > > > > > > I'm > > > > > > > > > > > > > > > > > > > > > > > > > > optimistic that we can > > > achieve > > > > > > this. > > > > > > > > > Hence, > > > > > > > > > > > +1 > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > contributing > > > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > back > > > > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > > Flink. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at > 2:03 > > > AM > > > > > > Sijie > > > > > > > > Guo > > > > > > > > > < > > > > > > > > > > > > > > > > > > guosi...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi Yun, > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> Since I was the main > > driver > > > > > behind > > > > > > > > > > > FLINK-9641 > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > FLINK-9168, > > > > > > > > > > > > > > > > > > > > > let > > > > > > > > > > > > > > > > > > > > > > me > > > > > > > > > > > > > > > > > > > > > > > > > try to > > > > > > > > > > > > > > > > > > > > > > > > > >> add more context on > this. > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and > FLINK-9168 > > > was > > > > > > > created > > > > > > > > > for > > > > > > > > > > > > > bringing > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > > source > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > >> sink for Flink. The > > > > integration > > > > > > was > > > > > > > > done > > > > > > > > > > > with > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > 1.6.0. > > > > > > > > > > > > > > > > > > > We > > > > > > > > > > > > > > > > > > > > > > sent out > > > > > > > > > > > > > > > > > > > > > > > > > pull > > > > > > > > > > > > > > > > > > > > > > > > > >> requests about a year > ago > > > and > > > > we > > > > > > > ended > > > > > > > > > up > > > > > > > > > > > > > > > maintaining > > > > > > > > > > > > > > > > > > those > > > > > > > > > > > > > > > > > > > > > > connectors > > > > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar for Pulsar users > to > > > use > > > > > > Flink > > > > > > > > to > > > > > > > > > > > > process > > > > > > > > > > > > > > > event > > > > > > > > > > > > > > > > > > > streams > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > Pulsar. > > > > > > > > > > > > > > > > > > > > > > > > > >> (See > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink > > > > > > > > > > > > > > > > > > > > > ). > > > > > > > > > > > > > > > > > > > > > > The > > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > >> 1.6 integration is > pretty > > > > simple > > > > > > and > > > > > > > > > there > > > > > > > > > > > is > > > > > > > > > > > > no > > > > > > > > > > > > > > > schema > > > > > > > > > > > > > > > > > > > > > > > > considerations. > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> In the past year, we > have > > > > made a > > > > > > lot > > > > > > > > of > > > > > > > > > > > > changes > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > brought > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar schema as the > > > > first-class > > > > > > > > > citizen in > > > > > > > > > > > > > > Pulsar. > > > > > > > > > > > > > > > We > > > > > > > > > > > > > > > > > > also > > > > > > > > > > > > > > > > > > > > > > integrated > > > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > > >> other computing engines > > for > > > > > > > processing > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > event > > > > > > > > > > > > > > > > > > streams > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > > >> schema. > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> It led us to rethink how > > to > > > > > > > integrate > > > > > > > > > with > > > > > > > > > > > > Flink > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > best > > > > > > > > > > > > > > > > > > > > > way. > > > > > > > > > > > > > > > > > > > > > > Then > > > > > > > > > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > > > > > > >> reimplement the > > pulsar-flink > > > > > > > > connectors > > > > > > > > > from > > > > > > > > > > > > the > > > > > > > > > > > > > > > ground > > > > > > > > > > > > > > > > > up > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > schema > > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > >> bring table API and > > catalog > > > > API > > > > > as > > > > > > > the > > > > > > > > > > > > > first-class > > > > > > > > > > > > > > > > > citizen > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > >> integration. With that > > being > > > > > said, > > > > > > > in > > > > > > > > > the > > > > > > > > > > > new > > > > > > > > > > > > > > > > > pulsar-flink > > > > > > > > > > > > > > > > > > > > > > > > > implementation, > > > > > > > > > > > > > > > > > > > > > > > > > >> you can register pulsar > > as a > > > > > flink > > > > > > > > > catalog > > > > > > > > > > > and > > > > > > > > > > > > > > > query / > > > > > > > > > > > > > > > > > > > process > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > event > > > > > > > > > > > > > > > > > > > > > > > > > >> streams using Flink SQL. > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> This is an example about > > how > > > > to > > > > > > use > > > > > > > > > Pulsar > > > > > > > > > > > as > > > > > > > > > > > > a > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > catalog: > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> Yijie has also written a > > > blog > > > > > post > > > > > > > > > > > explaining > > > > > > > > > > > > > why > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > > re-implement > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > flink > > > > > > > > > > > > > > > > > > > > > > > > > >> connector with Flink 1.9 > > and > > > > > what > > > > > > > are > > > > > > > > > the > > > > > > > > > > > > > changes > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > made > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > > > > > > > >> connector: > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> We believe Pulsar is not > > > just > > > > a > > > > > > > simple > > > > > > > > > data > > > > > > > > > > > > sink > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > source > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > Flink. > > > > > > > > > > > > > > > > > > > > > > > > It > > > > > > > > > > > > > > > > > > > > > > > > > >> actually can be a fully > > > > > integrated > > > > > > > > > streaming > > > > > > > > > > > > > data > > > > > > > > > > > > > > > > > storage > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > Flink in > > > > > > > > > > > > > > > > > > > > > > > > > many > > > > > > > > > > > > > > > > > > > > > > > > > >> areas (sink, source, > > > > > > schema/catalog > > > > > > > > and > > > > > > > > > > > > state). > > > > > > > > > > > > > > The > > > > > > > > > > > > > > > > > > > > combination > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > >> and Pulsar can create a > > > great > > > > > > > > streaming > > > > > > > > > > > > > warehouse > > > > > > > > > > > > > > > > > > > architecture > > > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > > > > >> streaming-first, unified > > > data > > > > > > > > > processing. > > > > > > > > > > > > Since > > > > > > > > > > > > > we > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > talking > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > >> contribute Pulsar > > > integration > > > > to > > > > > > > Flink > > > > > > > > > here, > > > > > > > > > > > > we > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > also > > > > > > > > > > > > > > > > > > > > > > dedicated to > > > > > > > > > > > > > > > > > > > > > > > > > >> maintain, improve and > > evolve > > > > the > > > > > > > > > integration > > > > > > > > > > > > > with > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > help > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > users > > > > > > > > > > > > > > > > > > > > > > > > > >> who use both Flink and > > > Pulsar. > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> Hope this give you a bit > > > more > > > > > > > > background > > > > > > > > > > > about > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > pulsar > > > > > > > > > > > > > > > > > > > > flink > > > > > > > > > > > > > > > > > > > > > > > > > >> integration. Let me know > > > what > > > > > are > > > > > > > your > > > > > > > > > > > > thoughts. > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > >> Sijie > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019 at > > 11:54 > > > > AM > > > > > > Yun > > > > > > > > > Tang < > > > > > > > > > > > > > > > > > > myas...@live.com> > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > >>> I can see that Pulsar > > > becomes > > > > > > more > > > > > > > > and > > > > > > > > > more > > > > > > > > > > > > > > popular > > > > > > > > > > > > > > > > > > > recently > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > very > > > > > > > > > > > > > > > > > > > > > > > > > >> glad > > > > > > > > > > > > > > > > > > > > > > > > > >>> to see more people > > willing > > > to > > > > > > > > > contribute to > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > ecosystem. > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > >>> Before any further > > > > discussion, > > > > > > > would > > > > > > > > > you > > > > > > > > > > > > please > > > > > > > > > > > > > > > give > > > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > > > > > explanation > > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > >>> the relationship > between > > > this > > > > > > > thread > > > > > > > > to > > > > > > > > > > > > current > > > > > > > > > > > > > > > > > existing > > > > > > > > > > > > > > > > > > > > JIRAs > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > pulsar > > > > > > > > > > > > > > > > > > > > > > > > > >>> source [1] and sink [2] > > > > > > connector? > > > > > > > > > Will the > > > > > > > > > > > > > > > > > contribution > > > > > > > > > > > > > > > > > > > > > contains > > > > > > > > > > > > > > > > > > > > > > > > part > > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > >>> those PRs or totally > > > > different > > > > > > > > > > > > implementation? > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1] > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9641 > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2] > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9168 > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > ________________________________ > > > > > > > > > > > > > > > > > > > > > > > > > >>> From: Yijie Shen < > > > > > > > > > > > henry.yijies...@gmail.com> > > > > > > > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday, > September > > 3, > > > > > 2019 > > > > > > > > 13:57 > > > > > > > > > > > > > > > > > > > > > > > > > >>> To: > dev@flink.apache.org > > < > > > > > > > > > > > > dev@flink.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Subject: [DISCUSS] > > > Contribute > > > > > > > Pulsar > > > > > > > > > Flink > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > back > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > >>> Dear Flink Community! > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > >>> I would like to open > the > > > > > > discussion > > > > > > > > of > > > > > > > > > > > > > > contributing > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > >>> connector [0] back to > > > Flink. > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## A brief introduction > > to > > > > > Apache > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > >>> Apache Pulsar[1] is a > > > > > > multi-tenant, > > > > > > > > > > > > > > > high-performance > > > > > > > > > > > > > > > > > > > > > distributed > > > > > > > > > > > > > > > > > > > > > > > > > >>> pub-sub messaging > system. > > > > > Pulsar > > > > > > > > > includes > > > > > > > > > > > > > > multiple > > > > > > > > > > > > > > > > > > features > > > > > > > > > > > > > > > > > > > > > such > > > > > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > > > > > >>> native support for > > multiple > > > > > > > clusters > > > > > > > > > in a > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > instance, > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > > >>> seamless > geo-replication > > of > > > > > > > messages > > > > > > > > > across > > > > > > > > > > > > > > > clusters, > > > > > > > > > > > > > > > > > > very > > > > > > > > > > > > > > > > > > > > low > > > > > > > > > > > > > > > > > > > > > > > > publish > > > > > > > > > > > > > > > > > > > > > > > > > >>> and end-to-end latency, > > > > > seamless > > > > > > > > > > > scalability > > > > > > > > > > > > to > > > > > > > > > > > > > > > over a > > > > > > > > > > > > > > > > > > > > million > > > > > > > > > > > > > > > > > > > > > > > > topics, > > > > > > > > > > > > > > > > > > > > > > > > > >>> and guaranteed message > > > > delivery > > > > > > > with > > > > > > > > > > > > persistent > > > > > > > > > > > > > > > message > > > > > > > > > > > > > > > > > > > > storage > > > > > > > > > > > > > > > > > > > > > > > > > >>> provided by Apache > > > > BookKeeper. > > > > > > > > > Nowadays, > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > has > > > > > > > > > > > > > > > > > been > > > > > > > > > > > > > > > > > > > > > adopted > > > > > > > > > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > > > > > > > > >>> more and more > > companies[2]. > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## The status of Pulsar > > > Flink > > > > > > > > connector > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > >>> The Pulsar Flink > > connector > > > we > > > > > are > > > > > > > > > planning > > > > > > > > > > > to > > > > > > > > > > > > > > > > > contribute > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > built > > > > > > > > > > > > > > > > > > > > > > > > upon > > > > > > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and Pulsar > > > 2.4.0. > > > > > The > > > > > > > > main > > > > > > > > > > > > features > > > > > > > > > > > > > > > are: > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a streaming > > > > source > > > > > > with > > > > > > > > > > > > > exactly-once > > > > > > > > > > > > > > > > > > guarantee. > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Sink streaming > results > > to > > > > > > Pulsar > > > > > > > > with > > > > > > > > > > > > > > > at-least-once > > > > > > > > > > > > > > > > > > > > > semantics. > > > > > > > > > > > > > > > > > > > > > > (We > > > > > > > > > > > > > > > > > > > > > > > > > >>> would update this to > > > > > exactly-once > > > > > > > as > > > > > > > > > well > > > > > > > > > > > > when > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > gets > > > > > > > > > > > > > > > > > > > > all > > > > > > > > > > > > > > > > > > > > > > > > > >>> transaction features > > ready > > > in > > > > > its > > > > > > > > 2.5.0 > > > > > > > > > > > > > version) > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Build upon Flink new > > > Table > > > > > API > > > > > > > Type > > > > > > > > > > > system > > > > > > > > > > > > > > > > > > (FLIP-37[3]), > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > > > > > > >>> automatically > > (de)serialize > > > > > > > messages > > > > > > > > > with > > > > > > > > > > > the > > > > > > > > > > > > > > help > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > schema. > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Integrate with Flink > > new > > > > > > Catalog > > > > > > > > API > > > > > > > > > > > > > > > (FLIP-30[4]), > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > > > > > enables > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > >>> use of Pulsar topics as > > > > tables > > > > > in > > > > > > > > > Table API > > > > > > > > > > > > as > > > > > > > > > > > > > > > well as > > > > > > > > > > > > > > > > > > SQL > > > > > > > > > > > > > > > > > > > > > > client. > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## Reference > > > > > > > > > > > > > > > > > > > > > > > > > >>> [0] > > > > > > > > > > > > > https://github.com/streamnative/pulsar-flink > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1] > > > > https://pulsar.apache.org/ > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2] > > > > > > > > > > > https://pulsar.apache.org/en/powered-by/ > > > > > > > > > > > > > > > > > > > > > > > > > >>> [3] > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System > > > > > > > > > > > > > > > > > > > > > > > > > >>> [4] > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best, > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
