Consumer group metadata retention

I am writing an exactly-once Kafka sink for Apache Beam. In order to avoid duplicates due to retries, it stores a sequential id and producer signature in consumer group metadata, and commits it atomically with messages (using sendOffsetsToTransaction()). I have a couple of clarification questions

Re: Consumer blocks forever on unreachable ports

quick update: This can be unblocked with consumer.wakeup(). So my current work around is to run this in a separate thread and cancel it after a timeout. On Fri, Jun 30, 2017 at 11:14 AM, Raghu Angadi wrote: > Consumer blocks forever during initialization if the brokers are not > rea

Consumer blocks forever on unreachable ports

Consumer blocks forever during initialization if the brokers are not reachable. 'request.timeout.ms' does not seem to be effective. How are users expected to detect this and handle? e.g: $ ./kafka-console-consumer.sh --bootstrap-server google.com:9092 --topic queries This blocks forever since goo

Is there a way to fetch latest offset (0.9 consumer)

We implemented a Kafka connector for Google Dataflow (streaming). We manually assign partitions to each split. Dataflow SDK lets sources report their backlog, I didn't see any way to find out about latest offset using 0.9 consumer. O

Re: How can a producer avoid slow brokers?

ix the problem. However if the problem is just load then this > will probably make things worse. It is also a bit tricky to define what is > "slow" and have the user accurately configure that. It would be easy to > imagine a half-assed implementation causing more problems th

Re: How can a producer avoid slow brokers?

ight be useful if you have the producer being fed by scribe > aggregators. > > Thanks, > Neha > > > On Fri, Jan 4, 2013 at 9:04 AM, Raghu Angadi > wrote: > > > On Fri, Jan 4, 2013 at 8:39 AM, Jun Rao wrote: > > > > > Do you know why some of the brok

Re: How can a producer avoid slow brokers?

On Fri, Jan 4, 2013 at 8:39 AM, Jun Rao wrote: > Do you know why some of the brokers are much slower than others? We are currently running these in a shared environment, to make things worse these machines have single spindle. We have to put up with that until we move the brokers to dedicated h

How can a producer avoid slow brokers?

Producer distributes messages uniformly across the partitions. This does not work very well when some of the brokers are much slower than others. Is there a way to temporarily avoid such slow brokers? While async producers, I could avoid producers that have lot more messages in their internal que