kafka connect

Hello All, I have a few questions regarding kafka connect. I'd appreciate your replies. 1) Is there a kafka connector for listening to tcp sockets? 2) If there a protobuf converter that can deal with variable length messages? Meaning read the prefix that specifies the length and use the specifie

Re: Consumers re-consuming messages again after re-balance?

Thanks guys, was exactly `offsets.retention.minutes`. Figured out that `enable.auto.commit` was set to false in reality, somewhere deep in spring properties and that's what have been causing offsets removal when idle. On Mon, Jul 3, 2017 at 7:04 PM, Dmitriy Vsekhvalnov wrote: > Ouch, interes

Re: exception processing streams ..

Thanks! On Tue, Jul 4, 2017 at 10:28 PM, Damian Guy wrote: > Yes, System.exit(..) > streams.close(..) just attempts to stop any running stream threads. > > On Tue, 4 Jul 2017 at 17:49 Debasish Ghosh > wrote: > >> Hi Damien - >> >> Just 1 question .. by "terminate the process" do you mean System

Re: exception processing streams ..

Yes, System.exit(..) streams.close(..) just attempts to stop any running stream threads. On Tue, 4 Jul 2017 at 17:49 Debasish Ghosh wrote: > Hi Damien - > > Just 1 question .. by "terminate the process" do you mean System.exit(..) ? > Because streams.close() will not terminate the process - righ

Re: exception processing streams ..

Hi Damien - Just 1 question .. by "terminate the process" do you mean System.exit(..) ? Because streams.close() will not terminate the process - right ? regards. On Tue, Jul 4, 2017 at 9:36 PM, Debasish Ghosh wrote: > Hi Damian - > > I also thought so .. yes, I will add `KafkaStreams#setUncaug

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

Hi Edo, Thanks for reviewing the KIP. Modifying the default behavior of `--describe` was suggested in the related JIRA. We could poll the community to see whether they go for that option, or, as you suggested, introducing a new `--only-xxx` ( can't also think of a proper name right now :) ) op

Re: exception processing streams ..

Hi Damian - I also thought so .. yes, I will add `KafkaStreams# setUncaughtExceptionHandler(...)` and Mesos should restart the process .. Thanks for your prompt response .. regards. On Tue, Jul 4, 2017 at 9:30 PM, Damian Guy wrote: > Hi Debasish, > > It looks like it is possibly a bug in the K

Re: exception processing streams ..

Hi Debasish, It looks like it is possibly a bug in the Kafka Consumer code. In your streams app you probably want to add an UncaughtExceptionHandler, i.e, via `KafkaStreams#setUncaughtExceptionHandler(...)` and terminate the process when you receive an uncaught exception. I guess Mesos should auto

exception processing streams ..

Hi - I have been running a streaming application on some data set. Things usually run ok. Today I was trying to run the same application on Kafka (ver 0.10.2.1 on Scala 2.12) installed in a Mesos DC/OS cluster. After running for quite some time, I got the following exception .. Exception in threa

Distribution of logs over log.dirs

Cheers! I have a setup with 3 brokers having 27 log.dirs each. Setting up a topic with 27 partitions and 2 replication I would have expected every directory of log.dirs to feature <1 partition. This is not the case (the setup has been the same from the beginning; e.g. no log.dirs added later).

Re: [DISCUSS] Streams DSL/StateStore Refactoring

Hi Jan, Thanks very much for the input. On Tue, 4 Jul 2017 at 08:54 Jan Filipiak wrote: > Hi Damian, > > I do see your point of something needs to change. But I fully agree with > Gouzhang when he says. > --- > > But since this is a incompatibility change, and we are going to remove the > compa

Re: [DISCUSS] Streams DSL/StateStore Refactoring

Hi Damian, I do see your point of something needs to change. But I fully agree with Gouzhang when he says. --- But since this is a incompatibility change, and we are going to remove the compatibility annotations soon it means we only have one chance and we really have to make it right. I