Thanks for the added info. For the mean time we'll rely on the older
ConsumerOffsetChecker for our monitoring tools.
Thanks,
Cliff
On Fri, Jan 29, 2016 at 10:56 AM, Guozhang Wang wrote:
> Tao,
>
> You are right, ConsumerOffsetChecker can still get offsets from the offset
> manager in Kafka.
>
Tao,
You are right, ConsumerOffsetChecker can still get offsets from the offset
manager in Kafka.
Guozhang
On Thu, Jan 28, 2016 at 9:36 PM, tao xiao wrote:
> it first issues an offsetrequest to broker and check if offset is stored in
> broker if not it will queries zk
>
>
> https://github.com/
it first issues an offsetrequest to broker and check if offset is stored in
broker if not it will queries zk
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L171
On Fri, 29 Jan 2016 at 13:11 Guozhang Wang wrote:
> Tao,
>
> Hmm that is a bit
Tao,
Hmm that is a bit wired since ConsumerOffsetChecker itself does not talk to
brokers at all, but only through ZK.
Guozhang
On Thu, Jan 28, 2016 at 6:07 PM, tao xiao wrote:
> Guozhang,
>
> The old ConsumerOffsetChecker works for new consumer too with offset stored
> in Kafka. I tested it wi
Guozhang,
The old ConsumerOffsetChecker works for new consumer too with offset stored
in Kafka. I tested it with mirror maker with new consumer enabled. it is
able to show offset during mirror maker running and after its shutdown.
On Fri, 29 Jan 2016 at 06:34 Guozhang Wang wrote:
> Once the off
Once the offset is written to the log it is persistent and hence should
survive broker failures. And its retention policy is configurable.
It may be a bit misleading in saying "in-memory cache" in my previous
email: the brokers just keep the in-memory map of [group, partition] ->
latest_offset, wh
Hi Guozhang,
That looks like it might help but feels like there might be some gaps.
Would it be able to survive restarts of the kafka broker? How long would
it stay in the cache (and is that configurable)? If it expires from the
cache, what's the cache-miss operation look like? (yes, a lot of t
Hi Cliff,
Short answer to your question is it is just the current implementation.
The coordinator stores the offsets as messages in an internal topic and
also keeps the latest offset values in in-memory. It answers
ConsumerGroupRequest using its cached offset, and upon the consumer group
being re
Just following up on this concern. Is there a constraint that prevents
ConsumerGroupCommand from reporting offsets on a group if no members are
connected, or is this just the current implementation?
Thanks,
Cliff
On Mon, Jan 25, 2016 at 3:50 PM, Cliff Rhyne wrote:
> I'm running into a few chal
