Since offsets in Kafka 0.7x are just byte counts, you cannot know the
number of messages remaining to be processed (subtract your consumers
offsets from each partitions end offset). However, you can know the
number of bytes remaining. Knowing the average message size, you can
use that to make a rou
Once you have an offset, is it possible to know how many messages there are
from that point to the end? (or least for the particular topic partition
that you are requested data from?).
The idea is to get an idea how far behind the consumers are from the # of
messages coming in etc.
I'm guessing t
Not sure if anyone else ran into this problem in Jconsole. I rely on this tool
to see the number of messages sent by the producer and verifies the number of
messages to consume by the consumer. Sometimes I found the Jconsole does not
display all messages in a topic on the broker after I restar
Thanks. This makes sense.
Jamie
-Original Message-
From: Neha Narkhede [mailto:neha.narkh...@gmail.com]
Sent: Monday, January 28, 2013 5:27 PM
To: users@kafka.apache.org
Subject: Re: How do you keep track of offset in a partition
Jamie,
You need to use the getOffsetsBefore() API to ge
Jamie,
You need to use the getOffsetsBefore() API to get the earliest/latest
offset available on the broker for a particular partition.
Thanks,
Neha
On Mon, Jan 28, 2013 at 5:05 PM, Jamie Wang wrote:
> Hi,
>
> We are using 0.72 version of Kafka on Windows. I am wondering what is the
> right w
Danny,
Please can you file a bug to explain the issue ?
Thanks,
Neha
On Mon, Jan 28, 2013 at 8:57 AM, Danny Yeshurun wrote:
> No data on the second time. The partition is empty but starting from the
> second request the response is slow.
>
> On Mon, Jan 28, 2013 at 6:31 PM, Jun Rao wrote:
>
>
No data on the second time. The partition is empty but starting from the
second request the response is slow.
On Mon, Jan 28, 2013 at 6:31 PM, Jun Rao wrote:
> Not sure what's the causing the slow down. Is the second fetch geting more
> data?
>
> Also, if all those partitions are on the same bro
Not sure what's the causing the slow down. Is the second fetch geting more
data?
Also, if all those partitions are on the same broker, you could have
fetched all partitions' data in a single fetch request.
Thanks,
Jun
On Mon, Jan 28, 2013 at 6:02 AM, Danny Yeshurun wrote:
> Hi,
>
> I am using
Hi,
I am using Kafka 0.7.2
I have a consumer that tries to fetch messages from several partitions on
the same broker (and topic).
The consumers issues a fetch request per partition, if there are messages
to read, it fetches the first one and goes to the next partition to fetch
another message and
