Timestamps are stored in the message headers. You can use the tools in kafka/bin to dump those messages timestamps to check their value. My guess is that the producer is setting a specific timestamp that is wrong ?
Le mer. 29 avr. 2020 à 10:09, JP MB <jose.brandao1...@gmail.com> a écrit : > Hi, > Sorry guys for insisting on this but does anyone have an idea of how that > largestTime can be 0 ? > > Regards > > ---------- Forwarded message --------- > De: JP MB <jose.brandao1...@gmail.com> > Date: ter., 28 de abr. de 2020 às 15:36 > Subject: Kafka: Messages disappearing from topics, largestTime=0 > To: <users@kafka.apache.org> > > > Hi, > We have messages disappearing from topics on Apache Kafka with versions > 2.3, 2.4.0, 2.4.1 and 2.5.0. We noticed this when we make a rolling > deployment of our clusters and unfortunately it doesn't happen every time, > so it's very inconsistent. > > Sometimes we lose all messages inside a topic, other times we lose all > messages inside a partition. When this happens the following log is a > constant: > > [2020-04-27 10:36:40,386] INFO [Log partition=test-lost-messages-5, > dir=/var/kafkadata/data01/data] Deleting segments > List(LogSegment(baseOffset=6, size=728, > lastModifiedTime=1587978859000, largestTime=0)) (kafka.log.Log) > > There is also a previous log saying this segment hit the retention time > breach of 48 hours. In this example, the message was produced ~12 minutes > before the deployment. > > Notice, all messages that are wrongly deleted havelargestTime=0 and the > ones that are properly deleted have a valid timestamp in there. From what > we read from documentation and code it looks like the largestTime is used > to calculate if a given segment reached the time breach or not. > > Since we can observe this in multiple versions of Kafka, we think this > might be related to anything external to Kafka. E.g Zookeeper. > > Does anyone have any ideas of why this could be happening? > For the record, we are using Zookeeper 3.6.0. > -- *Nicolas Carlot* Lead dev | | nicolas.car...@chronopost.fr *Veuillez noter qu'à partir du 20 mai, le siège Chronopost déménage. La nouvelle adresse est : 3 boulevard Romain Rolland 75014 Paris* [image: Logo Chronopost] | chronopost.fr <http://www.chronopost.fr/> Suivez nous sur Facebook <https://fr-fr.facebook.com/chronopost> et Twitter <https://twitter.com/chronopost>. [image: DPD Group]
