Hi Flinkers, As discussed with @tzulitai at apache/flink#5845 <https://github.com/apache/flink/pull/5845>, I am starting a discussion thread about contributing flink pulsar connectors (including both source and sink connectors) from pulsar community to flink project. We'd like to see what are people's thoughts about this and how we can proceed for this.
For people who doesn't know about Apache Pulsar, here are some background: --- Apache Pulsar (incubating) <https://pulsar.incubator.apache.org/> is a distributed pub/sub messaging system, which provides very flexible messaging model - unifying traditional queuing (e.g. SQS, rabbitmq) and high-performance streaming (e.g. Kinesis, Kafka) into one pub/sub messaging model + api. It is backed by a scalable segment/log storage Apache BookKeeper, which provide unbounded stream storage for Pulsar. Because of its segment-centric architecture design, Pulsar provides compelling unbounded streaming data storage. It is good for both streaming and batch processing, which I believe it fits very well into Flink's data processing model. Besides that, pulsar has a lot of advanced features going on its upcoming 2.0 release, including built-in schema registry, topic compaction, regex subscription, and tiered storage <https://github.com/apache/incubator-pulsar/wiki/PIP-17:-Tiered-storage-for-Pulsar-topics> ... Pulsar was developed by Yahoo since 2012-ish and has been running on production for 4+ years, over 10+ data centers and processing/delivering billions of messages per day. It was open sourced at 2016. Since it is open sourced, it has been adopted by various companies. Nowadays, the pulsar slack channel discussion is very active and fast-growing. The community currently has about 15 committers. --- I happened to work with ZongYang (who is also a pulsar contributor) on developing pulsar connectors for flink to satisfy pulsar users requests. We would like to contribute the connector work to flink and continue the collaboration between flink and pulsar communities. From pulsar community perspective, we are also very committed to developing pulsar's ecosystem, and willing and dedicated to developing/maintaining flink pulsar connectors. Hope this email thread give you guys enough background of pulsar and clear some of the concerns that @tzulitai raised in the jira ticket / pull request. Looking forward to any feedback from pulsar community and deep collaboration between flink and pulsar community. Also /cc pulsar dev mailing list (d...@pulsar.incubator.apache.org). If there are any questions, pulsar devs can also help to answer. Thanks, Sijie
