But then I wonder, why such things are not mentioned anywhere in Kafka
configuration document? I relied on that setting and it caused us some
issues. If it is mentioned clearly then everyone will be aware. Could you
please point in right direction about reading timestamp of log message? I
will see
Retention time is a lower bound for how long it is guaranteed that data
will be stored. This guarantee work "one way" only. There is no
guarantee when data will be deleted after the bound passed.
However, client side, you can always check the record timestamp and just
drop older data that is still
Can anyone here help me please? I am at my wit's end. I now have
max.poll.records set to just 2. Still I am getting Auto offset commit
failed warning. Log file is getting full because of this warning. Session
timeout is 5 minutes, max.poll.interval.ms is 10 minutes.
On Wed, May 23, 2018 at 12:42 P
If the consumer is idle (not doing any commit) for more than a day, the
offsets will be wiped out.
Rf. offsets.retention.minutes property in Kafka documentation.
On Tue, May 29, 2018 at 9:49 AM, Shantanu Deshmukh
wrote:
> Which Kafka version?
>
> On Mon, May 28, 2018 at 9:09 PM Dinesh Subramani
Which Kafka version?
On Mon, May 28, 2018 at 9:09 PM Dinesh Subramanian <
dsubraman...@apptivo.co.in> wrote:
> Hi,
>
> Whenever we bounce the consumer in tomcat node, I am facing duplication.
> It is consumed from the beginning. I have this property in consumer
> "auto.offset.reset" = "earliest
Please help.
On Mon, May 28, 2018 at 5:18 PM Shantanu Deshmukh
wrote:
> I have a topic otp-sms. I want that retention of this topic should be 5
> minutes as OTPs are invalid post that amount of time. So I set
> retention.ms=30. However, this was not working. So reading more in
> depth in Ka
Are you seeing 1) duplicate messages stored in a Kafka topic partition or 2)
duplicate consumption and processing of a single message stored in a Kafka
topic?
If it’s #1 then you can turn on the idempotent producer feature to get Exactly
Once Semantics (EOS) while publishing.
If it’s #2 then y
Hi,
Whenever we bounce the consumer in tomcat node, I am facing duplication.
It is consumed from the beginning. I have this property in consumer
"auto.offset.reset" = "earliest". if it is new consumer means it will
consume from the beginning, but it is consumed from the beginning for the
consume
Hello,
I work on the version 0.11 of Kafka, 14.04.5 of Ubuntu and 8 of Java. I have a
bug when I want to use an asymmetric time window in the Kstream-Kstream join.
More detail on this page
https://stackoverflow.com/questions/50412784/incorrect-result-kstream-kstream-join-with-asymmetric-time-win
I have a topic otp-sms. I want that retention of this topic should be 5
minutes as OTPs are invalid post that amount of time. So I set
retention.ms=30.
However, this was not working. So reading more in depth in Kafka
configuration document I found another topic level setting that can be
tuned
Hi Team,
Running kafka & zookeeper in kubernetes cluster.
No of brokers : 3
No of partitions per topic : 3
creating topic with 3 partitions, and looks like all the partitions are up.
Below is the snapshot to confirm the same,
Topic:applestore PartitionCount:3 ReplicationFactor:3 Con
Duplication can happen if your producer or consumer are exiting uncleanly.
Like if producer just crashes before it receives ack from broker your logic
will fail to register that message got produced. And when it comes back up
it will try to send that batch again. Same with consumer, if it crashes
b
Hi,
Facing Duplication inconsistently while bouncing Kafka producer and
consumer in tomcat node. any help will be appreciated to find out the root
cause.
--
With Regards,
Karthick.K
13 matches
Mail list logo
