Re: Messages disappearing from Kafka Streams topology

2024-03-26 Thread Karsten Stöckmann
Hi, thanks for getting back. I'll try and illustrate the issue. I've got an input topic 'folderTopicName' fed by a database CDC system. Messages then pass a series of FK left joins and are eventually sent to an output topic like this ('agencies' and 'documents' being KTables): strea

Re: Messages disappearing from Kafka Streams topology

2024-03-26 Thread mangat rai
Hey Karsten, There could be several reasons this could happen. 1. Did you check the error logs? There are several reasons why the Kafka stream app may drop incoming messages. Use exactly-once semantics to limit such cases. 2. Are you sure there was no error when deserializing the records from `fol

Re: Messages disappearing from Kafka Streams topology

2024-03-26 Thread Karsten Stöckmann
Hi Mangat, thanks for your thoughts. I had actually considered exactly-once semantics already, was unsure whether it would help, and left it aside for once then. I'll try that immediately when I get back to work. About snapshots and deserialization - I doubt that the issue is caused by deserializ

Re: Messages disappearing from Kafka Streams topology

2024-03-26 Thread mangat rai
Hey Karsten, So if a topic has not been created yet. Streams app will keep the data in memory, and then write it later when it is available. if your app is restarted (or thread is killed), you may lose data but it depends if the app will commit in the source topics. If there is no errors, then it