Seeing strong support from the Pulsar community and Arvid volunteering to mentor this effort resolved my concerns. I think the Pulsar connector will be a good addition to the project as it opens up more integrations.
Cheers, Till On Wed, Jan 6, 2021 at 11:50 AM Arvid Heise <ar...@ververica.com> wrote: > Hi Till, > > 1) Who from the Flink community will mentor this effort and could take > > responsibility for it? > > > > I'd be happy to mentor the transition. It remains to be seen who is doing > mainly the maintenance in the long run. If all fails, I can also take that > over but I was hoping that the respective connector authors would also > become Flink contributors/committers in the long run. > > 2) How can Pulsar be tested together with Flink (is there a Pulsar mini > >> cluster for IT tests)? > >> > > > > We use testcontainer for running a Pulsar standalone cluster to do the > > integration tests. If it is not suitable for Flink project, we are happy > to > > adopt any technologies the Flink community prefers. > > > > We currently try to establish testcontainers for some external systems, so > this sounds perfect to me. > > 3) Which Pulsar versions will/can be supported? > >> > > > > Pulsar has a pretty good backward compatibility between versions. In our > > experience, people have been running the connector with Pulsar clusters > > from 2.5.x to 2.7.x in production. > > > > I must admit that I didn't think too much about versions yet. Pulsar has a > fast development pace, so I'm assuming that we also need to cut support for > older versions rather quickly. We should avoid supporting several major > versions like we do with other connectors. Maybe we can phase out old > versions into flink-packages.org? > > If the Pulsar community is willing to help with some of these tasks, then > > this should be much easier but we need buy in from someone in the Flink > > community to help with this effort. > > > > I hope that Sijie and I addressed your concerns. > > Best, > > Arvid > > > On Wed, Jan 6, 2021 at 11:28 AM Sijie Guo <s...@streamnative.io> wrote: > > > Hi Till, > > > > Thank you for your email! Please find my comments inline. > > > > On Mon, Dec 28, 2020 at 5:50 AM Till Rohrmann <trohrm...@apache.org> > > wrote: > > > >> Hi Jianyun, > >> > >> Thanks a lot for reviving this discussion. I think it would be great to > >> have a well working Pulsar connector for Flink. Before diving into the > >> detailed plan on how to do it technically, I think we should discuss > where > >> exactly the connector should live. From the Flink community's > perspective > >> every additional connector adds a considerable maintenance burden and, > >> thus, needs to be carefully considered. > >> > >> 1) Who from the Flink community will mentor this effort and could take > >> responsibility for it? > >> > > > > We had a conversation with Stephen and Arvid about this. I think Arvid > > Heise was willing to mentor this effort. > > > > > >> 2) How can Pulsar be tested together with Flink (is there a Pulsar mini > >> cluster for IT tests)? > >> > > > > We use testcontainer for running a Pulsar standalone cluster to do the > > integration tests. If it is not suitable for Flink project, we are happy > to > > adopt any technologies the Flink community prefers. > > > > > >> 2.1) Do we need additional e2e tests? > >> > > > > The Pulsar-Flink connector repository contains pretty good test coverage. > > If it is not sufficient, we are happy to add more to follow Flink's best > > practices. > > > > > >> 3) Which Pulsar versions will/can be supported? > >> > > > > Pulsar has a pretty good backward compatibility between versions. In our > > experience, people have been running the connector with Pulsar clusters > > from 2.5.x to 2.7.x in production. > > > > > >> > >> If the Pulsar community is willing to help with some of these tasks, > then > >> this should be much easier but we need buy in from someone in the Flink > >> community to help with this effort. > >> > > > > I am one of the Pulsar PMC members. Some of our Pulsar PMC members and > > committers are committed to helping maintain this Pulsar Flink connector. > > Hope this helps eliminate some concerns here. > > > >> > >> The Flink community might be a bit unresponsive in the next couple of > >> days because of Christmas and New Year. Beginning of January the > discussion > >> should see a couple of additional comments. > >> > >> Cheers, > >> Till > >> > >> On Wed, Dec 23, 2020 at 10:37 AM 赵 建云 <zhaojianyu...@outlook.com> > wrote: > >> > >>> We are now done with the Flink Pulsar connector. It is ready to be > >>> merged. > >>> > >>> Now our plan is > >>> > >>> 1. merge Source and Sink based on Function, because it has been > >>> verified by users and is reliable. > >>> 2. Next is to merge the DynamicTable feature. table implementation > >>> depends on Step 1. > >>> 3. merge Pulsar Catalog > >>> 4. merge the PulsarSource based on FLIP-27: Refactor Source > Interface< > >>> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface > > > >>> of PulsarSource > >>> 5. merge PulsarSink based on FLIP-143: Unified Sink API< > >>> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API > >>> > > >>> 6. merge upsert-pulsar > >>> > >>> About wiki, I need to update the FLIP-72: Introduce Pulsar Connector< > >>> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-72%3A+Introduce+Pulsar+Connector > >, > >>> but I don't have access to it. > >>> > >>> StreamNative/Pulsar Flink Connector: > >>> https://github.com/streamnative/pulsar-flink > >>> > >>> > >>> Looking forward to your feedback~ > >>> > >>> Jianyun Zhao > >>> > >>> > > -- > > Arvid Heise | Senior Java Developer > > <https://www.ververica.com/> > > Follow us @VervericaData > > -- > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > Conference > > Stream Processing | Event Driven | Real Time > > -- > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany > > -- > Ververica GmbH > Registered at Amtsgericht Charlottenburg: HRB 158244 B > Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji > (Toni) Cheng >
