Hi Till, Thank you for your email! Please find my comments inline.
On Mon, Dec 28, 2020 at 5:50 AM Till Rohrmann <trohrm...@apache.org> wrote: > Hi Jianyun, > > Thanks a lot for reviving this discussion. I think it would be great to > have a well working Pulsar connector for Flink. Before diving into the > detailed plan on how to do it technically, I think we should discuss where > exactly the connector should live. From the Flink community's perspective > every additional connector adds a considerable maintenance burden and, > thus, needs to be carefully considered. > > 1) Who from the Flink community will mentor this effort and could take > responsibility for it? > We had a conversation with Stephen and Arvid about this. I think Arvid Heise was willing to mentor this effort. > 2) How can Pulsar be tested together with Flink (is there a Pulsar mini > cluster for IT tests)? > We use testcontainer for running a Pulsar standalone cluster to do the integration tests. If it is not suitable for Flink project, we are happy to adopt any technologies the Flink community prefers. > 2.1) Do we need additional e2e tests? > The Pulsar-Flink connector repository contains pretty good test coverage. If it is not sufficient, we are happy to add more to follow Flink's best practices. > 3) Which Pulsar versions will/can be supported? > Pulsar has a pretty good backward compatibility between versions. In our experience, people have been running the connector with Pulsar clusters from 2.5.x to 2.7.x in production. > > If the Pulsar community is willing to help with some of these tasks, then > this should be much easier but we need buy in from someone in the Flink > community to help with this effort. > I am one of the Pulsar PMC members. Some of our Pulsar PMC members and committers are committed to helping maintain this Pulsar Flink connector. Hope this helps eliminate some concerns here. > > The Flink community might be a bit unresponsive in the next couple of days > because of Christmas and New Year. Beginning of January the discussion > should see a couple of additional comments. > > Cheers, > Till > > On Wed, Dec 23, 2020 at 10:37 AM 赵 建云 <zhaojianyu...@outlook.com> wrote: > >> We are now done with the Flink Pulsar connector. It is ready to be merged. >> >> Now our plan is >> >> 1. merge Source and Sink based on Function, because it has been >> verified by users and is reliable. >> 2. Next is to merge the DynamicTable feature. table implementation >> depends on Step 1. >> 3. merge Pulsar Catalog >> 4. merge the PulsarSource based on FLIP-27: Refactor Source Interface< >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface> >> of PulsarSource >> 5. merge PulsarSink based on FLIP-143: Unified Sink API< >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API >> > >> 6. merge upsert-pulsar >> >> About wiki, I need to update the FLIP-72: Introduce Pulsar Connector< >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-72%3A+Introduce+Pulsar+Connector>, >> but I don't have access to it. >> >> StreamNative/Pulsar Flink Connector: >> https://github.com/streamnative/pulsar-flink >> >> >> Looking forward to your feedback~ >> >> Jianyun Zhao >> >>
