Hi,
We have messages disappearing from topics on Apache Kafka with versions
2.3, 2.4.0, 2.4.1 and 2.5.0. We noticed this when we make a rolling
deployment of our clusters and unfortunately it doesn't happen every time,
so it's very inconsistent.
Sometimes we lose all messages inside a topic, other
Hi,
Sorry guys for insisting on this but does anyone have an idea of how that
largestTime can be 0 ?
Regards
-- Forwarded message -
De: JP MB
Date: ter., 28 de abr. de 2020 às 15:36
Subject: Kafka: Messages disappearing from topics, largestTime=0
To:
Hi,
We have messages
We are using the console produce, directly on the machines where we are
experiencing the problem. I just inserted 150 messages in a topic and chose
the partition with more messages to make this analysis, in this case,
partition 15 in broker 1.
The log file:
> kafka-run-class.sh kafka.tools.DumpLo
; record is empty.
>
> On Tuesday, April 28, 2020, 04:37:03 PM GMT+2, JP MB <
> jose.brandao1...@gmail.com> wrote:
>
> Hi,
> We have messages disappearing from topics on Apache Kafka with versions
> 2.3, 2.4.0, 2.4.1 and 2.5.0. We noticed this when we make a rolling
&
rlot <
nicolas.car...@chronopost.fr> escreveu:
> Can you try using the console consumer to display messages/keys and
> timestamps ?
> --property print.key=true --property print.timestamp=true
>
>
> Le mer. 29 avr. 2020 à 13:23, JP MB a écrit :
>
> > The server
an idea of what could
be corrupting those index files.
Next things I will check:
- When exactly those messages were deleted in partitions 11 and 14.
- What happens if I have timeindex files with a "timestamp mismatch
error" and manually start and stop a broker.
Once aga
KillSignal is SIGTERM).
>
> Also, the Kafka broker logs at shutdown look very different if it shut down
> currently vs if it didn't. Could you perhaps put them in a Gist and email
> the link?
>
> Just trying to make sure basic assumptions are holding :)
>
> On Fri, 1 May 2
ching
> and do a test roll out, see if you get the same problems or not, would help
> eliminate it or confirm it.
>
> Sorry I can't help further on that.
>
> On Fri, May 1, 2020 at 5:34 AM JP MB wrote:
>
> > I took a bit because I needed logs of the server shutting down
> > possible there's some journalled but not flushed changes in those time
> > > indexes, but at this point we're getting into filesystem internals
> which
> > > aren't my forte. But if you can temporarily disable the volume
> switching
> > &
.timeindex files state before the deployment:
*All partitions without messages: permission denied, new files were created.
Don't see anything particular in the logs but you can see that the messages
are deleted with largestTime=0 and are from today morning.
Em seg., 4 de mai. de 2020 às 11:37,
020 às 08:52, Nicolas Carlot
escreveu:
> Hi guys, still following your discussion even if it's out of my reach.
> Just been noticing that you use /tmp/ for your logs, dunno if it's a good
> idea :o https://issues.apache.org/jira/browse/KAFKA-3925
>
> Le lun. 4 mai 2020 à
?
Best regards
Em ter., 5 de mai. de 2020 às 10:13, JP MB
escreveu:
> Hi,
> What I can see from the configurations:
>
>> log.dir = /tmp/kafka-logs (default)
>> log.dirs = /var/kafkadata/data01/data
>
>
> From the documentation log.dir is only used if log.dirs is not set
Hi guys,
My use case is simply copying data from one Kafka to another. When
searching on google, the immediate answer seems to be Mirror Maker, so we
jumped to the most recent version MM2.
The thing is I don't want active/active replication, the consumers from
cluster A will be different from the
;consume from topic A in cluster
> X, produce to topic A in cluster Y" app.
>
> On Fri, 15 May 2020, 12:13 am JP MB, wrote:
>
> > Hi guys,
> > My use case is simply copying data from one Kafka to another. When
> > searching on google, the immediate answer seems t
14 matches
Mail list logo
