Hi Jianyun, Thanks a lot for reviving this discussion. I think it would be great to have a well working Pulsar connector for Flink. Before diving into the detailed plan on how to do it technically, I think we should discuss where exactly the connector should live. From the Flink community's perspective every additional connector adds a considerable maintenance burden and, thus, needs to be carefully considered.
1) Who from the Flink community will mentor this effort and could take responsibility for it? 2) How can Pulsar be tested together with Flink (is there a Pulsar mini cluster for IT tests)? 2.1) Do we need additional e2e tests? 3) Which Pulsar versions will/can be supported? If the Pulsar community is willing to help with some of these tasks, then this should be much easier but we need buy in from someone in the Flink community to help with this effort. The Flink community might be a bit unresponsive in the next couple of days because of Christmas and New Year. Beginning of January the discussion should see a couple of additional comments. Cheers, Till On Wed, Dec 23, 2020 at 10:37 AM 赵 建云 <zhaojianyu...@outlook.com> wrote: > We are now done with the Flink Pulsar connector. It is ready to be merged. > > Now our plan is > > 1. merge Source and Sink based on Function, because it has been > verified by users and is reliable. > 2. Next is to merge the DynamicTable feature. table implementation > depends on Step 1. > 3. merge Pulsar Catalog > 4. merge the PulsarSource based on FLIP-27: Refactor Source Interface< > https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface> > of PulsarSource > 5. merge PulsarSink based on FLIP-143: Unified Sink API< > https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API > > > 6. merge upsert-pulsar > > About wiki, I need to update the FLIP-72: Introduce Pulsar Connector< > https://cwiki.apache.org/confluence/display/FLINK/FLIP-72%3A+Introduce+Pulsar+Connector>, > but I don't have access to it. > > StreamNative/Pulsar Flink Connector: > https://github.com/streamnative/pulsar-flink > > > Looking forward to your feedback~ > > Jianyun Zhao > >
