Hi Yijie I can see that Pulsar becomes more and more popular recently and very glad to see more people willing to contribute to Flink ecosystem.
Before any further discussion, would you please give some explanation of the relationship between this thread to current existing JIRAs of pulsar source [1] and sink [2] connector? Will the contribution contains part of those PRs or totally different implementation? [1] https://issues.apache.org/jira/browse/FLINK-9641 [2] https://issues.apache.org/jira/browse/FLINK-9168 Best Yun Tang ________________________________ From: Yijie Shen <henry.yijies...@gmail.com> Sent: Tuesday, September 3, 2019 13:57 To: dev@flink.apache.org <dev@flink.apache.org> Subject: [DISCUSS] Contribute Pulsar Flink connector back to Flink Dear Flink Community! I would like to open the discussion of contributing Pulsar Flink connector [0] back to Flink. ## A brief introduction to Apache Pulsar Apache Pulsar[1] is a multi-tenant, high-performance distributed pub-sub messaging system. Pulsar includes multiple features such as native support for multiple clusters in a Pulsar instance, with seamless geo-replication of messages across clusters, very low publish and end-to-end latency, seamless scalability to over a million topics, and guaranteed message delivery with persistent message storage provided by Apache BookKeeper. Nowadays, Pulsar has been adopted by more and more companies[2]. ## The status of Pulsar Flink connector The Pulsar Flink connector we are planning to contribute is built upon Flink 1.9.0 and Pulsar 2.4.0. The main features are: - Pulsar as a streaming source with exactly-once guarantee. - Sink streaming results to Pulsar with at-least-once semantics. (We would update this to exactly-once as well when Pulsar gets all transaction features ready in its 2.5.0 version) - Build upon Flink new Table API Type system (FLIP-37[3]), and can automatically (de)serialize messages with the help of Pulsar schema. - Integrate with Flink new Catalog API (FLIP-30[4]), which enables the use of Pulsar topics as tables in Table API as well as SQL client. ## Reference [0] https://github.com/streamnative/pulsar-flink [1] https://pulsar.apache.org/ [2] https://pulsar.apache.org/en/powered-by/ [3] https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System [4] https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs Best, Yijie Shen
