Hi Senthil,
Yes, I believe the only guarantee is that the offsets will be increasing
but not guaranteed to be consecutive.
Regards,
Srinath.
On Mon, Jul 1, 2019 at 4:15 PM SenthilKumar K
wrote:
> Hello Experts, We are trying to understand *"How Kafka Assign Offset value
> to a message?"*
>
>
Hello Experts, We are trying to understand *"How Kafka Assign Offset value
to a message?"*
Kafka Version : 2.2.0
Kafka Client: 1.1.0
For a Topic , Using Java Consumer API we consumed data from Partition 0 (
total 48 Partitions ) and below are the offset numbers.
181933
181935
181936
181939
r bad for kafka I know..) it seems we are limiting
ourselves severely by using kafka offsets.
Any more insight is still welcome of course.
Forwarded Message
Subject: unknown (kafka) offsets after restart
Date: Fri, 6 May 2016 14:12:24 +0200
From: Jörg Wagner
Rep
into this we found out that the Offsetmanager is
single threaded and due to our setup (few, powerful servers: rather bad for
kafka I know..) it seems we are limiting ourselves severely by using kafka
offsets.
Any more insight is still welcome of course.
Forwarded Message
Su
r
> kafka I know..) it seems we are limiting ourselves severely by using kafka
> offsets.
>
> Any more insight is still welcome of course.
>
>
> Forwarded Message ----
> Subject: unknown (kafka) offsets after restart
> Date: Fri, 6 May 2016 14:
After a bit more looking into this we found out that the Offsetmanager
is single threaded and due to our setup (few, powerful servers: rather
bad for kafka I know..) it seems we are limiting ourselves severely by
using kafka offsets.
Any more insight is still welcome of course
We're using Kafka 0.8.2 and are puzzled by the offset behaviour when
they are stored in kafka topics.
Upon restart of the Kafka cluster (e.g. due to reconfiguration) it can
happen that the offsets are unknown and therefore stop consumers from
consuming without knowing their offset.
kafka-run
ig retention setting. As you said, it won't
> necessarily be 0.
>
> Sending the same set of messages repeatedly resulted in having a very large
> count in the offset - a count that bore no relation to the number of
> messages in the topic - which worried me because I couldn'
t - a count that bore no relation to the number of
messages in the topic - which worried me because I couldn't explain it --
and things I can't explain make me nervous in the context of disaster
recovery...
I appreciate your confirmation of my theory about what is going on.
--JohnB (aka s
Hi John,
Kafka offsets are sequential id numbers that identify messages in each
partition. It might not be sequential within a topic (which can have
multiple partition).
Offsets don't necessarily start at 0 since messages are deleted.
.bin/kafka-run-class.sh kafka.tools.GetOffsetShell is p
Hmmm... more info.
So, inside /var/log/kafka-logs/myTopicName-0 I find two files
00026524.index 00026524.log
Interestingly, they both bear the number of the "lowest" offset returned by
the command I mention above.
If I "cat" the 000.26524.log file, I get all my mes
Thank you Christian -- I appreciate your taking the time to help me out on
this.
Here's what I found while continuing to dig into this.
If I take 30024 and subtract the number of messages I know I have in Kafka
(3500) I get 26524.
If I reset thus: set /kafka/consumers/myGroupName/offsets/myTopi
The number is the log-ordered number of bytes. So really, the offset is
kinda like the "number of bytes" to begin reading from. 0 means read the
log from the beginning. The second message is 0 + size of message. So the
message "ids" are really just the offset of the previous message sizes.
For exa
*Use Case: Disaster Recovery & Re-indexing SOLR*
I'm using Kafka to hold messages from a service that prepares "documents"
for SOLR.
A second micro service (a consumer) requests these messages, does any final
processing, and fires them into SOLR.
The whole thing is (in part) designed to be used
14 matches
Mail list logo
