Re: The household of the Kafka connector

2015-06-23 Thread Stephan Ewen
May bad, (4) is clear now. On Mon, Jun 22, 2015 at 8:59 PM, Aljoscha Krettek wrote: > Marton referred to the KafkaSink in 4). For sources the job will keep > running by reading from a different broker. > > On Mon, 22 Jun 2015 at 18:45 Stephan Ewen wrote: > > > I would like to consolidate those

Re: The household of the Kafka connector

2015-06-22 Thread Aljoscha Krettek
Marton referred to the KafkaSink in 4). For sources the job will keep running by reading from a different broker. On Mon, 22 Jun 2015 at 18:45 Stephan Ewen wrote: > I would like to consolidate those as well. > > Biggest blocker is, however, that the PersistentKafkaSource never commits > to zooke

Re: The household of the Kafka connector

2015-06-22 Thread Stephan Ewen
I would like to consolidate those as well. Biggest blocker is, however, that the PersistentKafkaSource never commits to zookeeper when checkpointing is not enabled. It should at least group commit periodically in those cases. Concerning (4), I though the high-level consumer (that we build the Per

The household of the Kafka connector

2015-06-22 Thread Márton Balassi
Hey, Due to the effort invested to the Kafka connector mainly by Robert and Gabor Hermann we are going to ship a fairly nice solution for reading from and writing to Kafka with 0.9.0. This is the most prominent streaming connector currently, and rightfully so as pipeline level end-to-end exactly o