The jmx should be of the form of clientId*-ConsumerLag under kafka.server.
Pausing the iteration will indirectly pause the underlying fetcher.
Thanks,
Jun
On Wed, Jun 11, 2014 at 3:09 AM, Bogdan Dimitriu (bdimitri) <
bdimi...@cisco.com> wrote:
> Which JMX MBeans are you referring to, Jun? I co
Which JMX MBeans are you referring to, Jun? I couldn’t find anything that
gives me the same information as the ConsumerOffsetChecker tool.
In any case, my main problem is that I don’t know when I should slow down
the iteration because I don’t know which stream the iteration is
consuming. I have the
We do have a jmx that reports the lag per partition. You could probably get
the lag that way. Then, you just need to slow down the iteration on the
fast partition.
Thanks,
Jun
On Mon, Jun 9, 2014 at 4:07 AM, Bogdan Dimitriu (bdimitri) <
bdimi...@cisco.com> wrote:
> Certainly.
> I know this may
Hi Bogdan,
It sounds as if you could implement a form of signaling between the
consumers using a distributed barrier. This can be implemented using Kafka
topics. For example you could create a control thread that posts the
current high-water mark for all consumers into a special topic, which give
Certainly.
I know this may not sound like a great idea but I am running out of
options here: I¹m basically trying to implement a consumer throttle. My
application consumes from a fairly high number of partitions from a number
of consumer servers. The data is put in the partitions by a producer in a
Could you elaborate on the use case of the stream ID?
Thanks,
Jun
On Fri, Jun 6, 2014 at 2:13 AM, Bogdan Dimitriu (bdimitri) <
bdimi...@cisco.com> wrote:
> Hello folks,
>
> I’m using Kafka 0.8.0 with the high level consumer and I have a situation
> where I need to obtain the ID for each of the
It is not easy to get the fetcher thread id from kafka stream. We do have
the mapping as a topicThreadIdAndQueues, but it is private to the consumer
and not exposed to callers.
Guozhang
On Fri, Jun 6, 2014 at 9:00 AM, Bogdan Dimitriu (bdimitri) <
bdimi...@cisco.com> wrote:
> I see. And can I so
I see. And can I somehow reliably get the ID of the fetcher thread that is
providing data for a KafkaStream? Specifically I¹d like to know that ID
from the thread where I¹m consuming (where I iterate through the stream).
Thank you,
Bogdan
On 06/06/2014 16:16, "Guozhang Wang" wrote:
>Bogdan,
>
>
Bogdan,
The kafka stream does not have an ID itself, the one you mentioned is the
ID of the fetcher thread that put data into the stream. Although there is a
one-to-one mapping between the fetcher thread and the stream, the Ids of
the fetcher cannot be accessed from the kafka stream itself.
Guozh
