Re: Querying consumer groups programmatically (from Golang)

2017-08-12 Thread Ian Duffy
Hi Jens, We did something similar to this at Zalando. https://github.com/zalando-incubator/remora It effectively supplies the kafka consumer group supply command as a http endpoint. On 12 August 2017 at 16:42, Subhash Sriram wrote: > Hi Jens, > > Have you looked at Burrow? > > https://github.

Re: How to debug - NETWORK_EXCEPTION

2017-08-12 Thread Raghav
Hey Martin I am using default setting for queue.enqueue.timeout.ms. since I have not set it my Java client. Network doesn't seem to time out either. Could I be missing something else ? On Sat, Aug 12, 2017 at 5:18 AM, Martin Gainty wrote: > > > > From: Raghav

Re: Forwarding consumer with kafka streams

2017-08-12 Thread Eno Thereska
Hi Ricardo, Kafka Streams should handle that case as well. What streams config are you using, could you share it? There is one parameter that is called “ConsumerConfig.AUTO_OFFSET_RESET_CONFIG” and by default it’s set to “earliest”. Any chance your app has changed it to “latest”? Thanks Eno >

Forwarding consumer with kafka streams

2017-08-12 Thread Ricardo Costa
Hi, I've implemented a forwarding consumer which literally just consumes the messages from a source topic, logs them and then publishes them to a target topic. I wanted to keep the implementation simple with very little code so I went with kafka-streams. I have a really simple topology with a sou

Re: Querying consumer groups programmatically (from Golang)

2017-08-12 Thread Subhash Sriram
Hi Jens, Have you looked at Burrow? https://github.com/linkedin/Burrow/blob/master/README.md Thanks, Subhash Sent from my iPhone > On Aug 12, 2017, at 8:55 AM, Jens Rantil wrote: > > Hi, > > I am one of the maintainers of prometheus-kafka-consumer-group-exporter[1], > which exports consumer

Querying consumer groups programmatically (from Golang)

2017-08-12 Thread Jens Rantil
Hi, I am one of the maintainers of prometheus-kafka-consumer-group-exporter[1], which exports consumer group offsets and lag to Prometheus. The way we currently scrape this information is by periodically executing `kafka-consumer-groups.sh --describe` for each group and parse the output. Recently

Re: How to debug - NETWORK_EXCEPTION

2017-08-12 Thread Martin Gainty
From: Raghav Sent: Thursday, August 10, 2017 12:51 AM To: Users; confluent-platf...@googlegroups.com Subject: How to debug - NETWORK_EXCEPTION Hi I am sending very small 32 byte message to Kafka broker in a tight loop with 250ms sleep. I have one broker, 1 part