Hi Fabian, Awesome, glad to hear back from flink community! Hope this is a good starting point for a tighter collaboration between two communities.
Thank you, Sijie On Tue, Apr 24, 2018 at 2:32 AM, Fabian Hueske <fhue...@gmail.com> wrote: > Hi Sijie, hi Pulsar community! > > Thanks for the detailed overview of Pulsar. > I like the idea of adding a Pulsar connector to Flink. > > As Gordon mentioned, the Flink community would like to ensure that the > connector is maintained after being added. > We experienced that connector maintenance, including fixing bugs, upgrading > to newer versions, adding new features, and reviewing contributions, > results in a lot of work for the community > Since you ensured that the Pulsar community is eager to help with this > effort, I think we could add the connector to Flink. > This would also be a good initiative for a tighter collaboration between > two ASF projects. > > Best, Fabian > > 2018-04-22 7:02 GMT+02:00 Henry Saputra <henry.sapu...@gmail.com>: > > > Here is the link to Apache Flink JIRA issue for this: > > > > https://issues.apache.org/jira/browse/FLINK-9168 > > > > - Henry > > > > On Fri, Apr 20, 2018 at 12:08 AM, Sijie Guo <guosi...@gmail.com> wrote: > > > > > Hi Flinkers, > > > > > > As discussed with @tzulitai at apache/flink#5845 > > > <https://github.com/apache/flink/pull/5845>, I am starting a > discussion > > > thread about contributing flink pulsar connectors (including both > source > > > and sink connectors) from pulsar community to flink project. We'd like > to > > > see what are people's thoughts about this and how we can proceed for > > this. > > > > > > For people who doesn't know about Apache Pulsar, here are some > > background: > > > > > > --- > > > > > > Apache Pulsar (incubating) <https://pulsar.incubator.apache.org/> is a > > > distributed pub/sub messaging system, which provides very flexible > > > messaging model - unifying traditional queuing (e.g. SQS, rabbitmq) and > > > high-performance streaming (e.g. Kinesis, Kafka) into one pub/sub > > messaging > > > model + api. It is backed by a scalable segment/log storage Apache > > > BookKeeper, which provide unbounded stream storage for Pulsar. Because > of > > > its segment-centric architecture design, Pulsar provides compelling > > > unbounded streaming data storage. It is good for both streaming and > batch > > > processing, which I believe it fits very well into Flink's data > > processing > > > model. Besides that, pulsar has a lot of advanced features going on its > > > upcoming 2.0 release, including built-in schema registry, topic > > compaction, > > > regex subscription, and tiered storage > > > <https://github.com/apache/incubator-pulsar/wiki/PIP-17:- > > > Tiered-storage-for-Pulsar-topics> > > > ... > > > > > > Pulsar was developed by Yahoo since 2012-ish and has been running on > > > production for 4+ years, over 10+ data centers and > processing/delivering > > > billions of messages per day. It was open sourced at 2016. Since it is > > open > > > sourced, it has been adopted by various companies. Nowadays, the pulsar > > > slack channel discussion is very active and fast-growing. The community > > > currently has about 15 committers. > > > > > > --- > > > > > > I happened to work with ZongYang (who is also a pulsar contributor) on > > > developing pulsar connectors for flink to satisfy pulsar users > requests. > > We > > > would like to contribute the connector work to flink and continue the > > > collaboration between flink and pulsar communities. From pulsar > community > > > perspective, we are also very committed to developing pulsar's > ecosystem, > > > and willing and dedicated to developing/maintaining flink pulsar > > > connectors. > > > > > > Hope this email thread give you guys enough background of pulsar and > > clear > > > some of the concerns that @tzulitai raised in the jira ticket / pull > > > request. Looking forward to any feedback from pulsar community and deep > > > collaboration between flink and pulsar community. > > > > > > Also /cc pulsar dev mailing list (d...@pulsar.incubator.apache.org). If > > > there are any questions, pulsar devs can also help to answer. > > > > > > Thanks, > > > Sijie > > > > > >
