The issue with producer metadata often occurs for repartition topics. Those are purged actively by Kafka Streams. It might help to increase the segment size and maybe `segment.ms` config for those topics to preserve a large history (the active segment is not purged).
By default, Kafka Streams creates those topics with smaller segment sized to make data purging more efficient. Not sure if your app has repartition topics though? -Matthias On 5/28/20 3:29 AM, Schmidt-Dumont Georg (BCI/ESW17) wrote: > Good morning, > > Since a couple of days ago we suddenly have the issue in our Kafka Steams > application that a UnknownProducerException occurs. Digging into this I came > across > KIP-360<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89068820>. > We are using a centrally managed Kafka, hence we are not able to just update > it to the latest version which contains the fix for the corresponding issues. > The Kafka version we are using is 2.2.1. > > I also found that a Workaround for Kafka versions pre 2.4 is to increase the > retention time and the transactional.id.expiration.ms. However, the retention > time for the topic for which the issue occurs already has a retention time of > 30 days. So from my point of view this cannot be the reason the meta data for > the producer was deleted. > > At this point I have two questions I hope someone can help me with. > > First, might be the reason that the producer meta data was deleted other than > all the data was removed from the topic because of the retention time and low > traffic on the topic? > > Second, the topic in question is used by a local state store. Since we > currently only have a single instance of the Kafka Streams application > running, I believe it would be safe to stop the application, flush the topic > and then start the app again. The state would be loaded from the file system > in this case and no data would be lost. Is this a valid assumption and would > the flush of the topic fix the issue? Or would we need to recreate the topic? > > Thanks! > > Best regards / Mit freundlichen Grüßen / Üdvözlettel / 致以诚挚的问候 > > Mr. Georg Schmidt-Dumont > Bosch Connected Industry – BCI/ESW17 > Robert Bosch GmbH | Postfach 10 60 50 | 70049 Stuttgart | GERMANY | > www.bosch.com<http://www.bosch.com/> > Phone +49 711 811-49893 | > georg.schmidt-dum...@bosch.com<mailto:georg.schmidt-dum...@bosch.com> > > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000; > Aufsichtsratsvorsitzender: Franz Fehrenbach; Geschäftsführung: Dr. Volkmar > Denner, > Prof. Dr. Stefan Asenkerschbaumer, Dr. Rolf Bulander, Dr. Stefan Hartung, Dr. > Markus Heyn, Dr. Dirk Hoheisel, > Christoph Kübel, Uwe Raschke, Peter Tyroller >
signature.asc
Description: OpenPGP digital signature
